Jul 09 15:29:00.985416 ip-10-0-133-183 systemd[1]: kubelet.service: Failed to load environment files: No such file or directory Jul 09 15:29:00.985425 ip-10-0-133-183 systemd[1]: kubelet.service: Failed to run 'start-pre' task: No such file or directory Jul 09 15:29:00.985432 ip-10-0-133-183 systemd[1]: kubelet.service: Failed with result 'resources'. Jul 09 15:29:00.985669 ip-10-0-133-183 systemd[1]: Failed to start Kubernetes Kubelet. Jul 09 15:29:11.202987 ip-10-0-133-183 systemd[1]: kubelet.service: Failed to schedule restart job: Unit crio.service not found. Jul 09 15:29:11.203002 ip-10-0-133-183 systemd[1]: kubelet.service: Failed with result 'resources'. -- Boot fd36e21d8d964a8a8c163756943956ad -- Jul 09 15:31:17.033297 ip-10-0-133-183 systemd[1]: Starting Kubernetes Kubelet... Jul 09 15:31:17.405875 ip-10-0-133-183 kubenswrapper[2573]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jul 09 15:31:17.405875 ip-10-0-133-183 kubenswrapper[2573]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version. Jul 09 15:31:17.405875 ip-10-0-133-183 kubenswrapper[2573]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jul 09 15:31:17.405875 ip-10-0-133-183 kubenswrapper[2573]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI. Jul 09 15:31:17.405875 ip-10-0-133-183 kubenswrapper[2573]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jul 09 15:31:17.406885 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:31:17.406738 2573 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime" Jul 09 15:31:17.411369 ip-10-0-133-183 kubenswrapper[2573]: W0709 15:31:17.411351 2573 feature_gate.go:328] unrecognized feature gate: Example2 Jul 09 15:31:17.411369 ip-10-0-133-183 kubenswrapper[2573]: W0709 15:31:17.411368 2573 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 09 15:31:17.411438 ip-10-0-133-183 kubenswrapper[2573]: W0709 15:31:17.411373 2573 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 09 15:31:17.411438 ip-10-0-133-183 kubenswrapper[2573]: W0709 15:31:17.411377 2573 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 09 15:31:17.411438 ip-10-0-133-183 kubenswrapper[2573]: W0709 15:31:17.411380 2573 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 09 15:31:17.411438 ip-10-0-133-183 kubenswrapper[2573]: W0709 15:31:17.411385 2573 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 09 15:31:17.411438 ip-10-0-133-183 kubenswrapper[2573]: W0709 15:31:17.411389 2573 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 09 15:31:17.411438 ip-10-0-133-183 kubenswrapper[2573]: W0709 15:31:17.411393 2573 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 09 15:31:17.411438 ip-10-0-133-183 kubenswrapper[2573]: W0709 15:31:17.411396 2573 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 09 15:31:17.411438 ip-10-0-133-183 kubenswrapper[2573]: W0709 15:31:17.411399 2573 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 09 15:31:17.411438 ip-10-0-133-183 kubenswrapper[2573]: W0709 15:31:17.411402 2573 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 09 15:31:17.411438 ip-10-0-133-183 kubenswrapper[2573]: W0709 15:31:17.411404 2573 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 09 15:31:17.411438 ip-10-0-133-183 kubenswrapper[2573]: W0709 15:31:17.411407 2573 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 09 15:31:17.411438 ip-10-0-133-183 kubenswrapper[2573]: W0709 15:31:17.411409 2573 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 09 15:31:17.411438 ip-10-0-133-183 kubenswrapper[2573]: W0709 15:31:17.411412 2573 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 09 15:31:17.411438 ip-10-0-133-183 kubenswrapper[2573]: W0709 15:31:17.411416 2573 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 09 15:31:17.411438 ip-10-0-133-183 kubenswrapper[2573]: W0709 15:31:17.411419 2573 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 09 15:31:17.411438 ip-10-0-133-183 kubenswrapper[2573]: W0709 15:31:17.411422 2573 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 09 15:31:17.411438 ip-10-0-133-183 kubenswrapper[2573]: W0709 15:31:17.411425 2573 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 09 15:31:17.411438 ip-10-0-133-183 kubenswrapper[2573]: W0709 15:31:17.411428 2573 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 09 15:31:17.411438 ip-10-0-133-183 kubenswrapper[2573]: W0709 15:31:17.411430 2573 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 09 15:31:17.411875 ip-10-0-133-183 kubenswrapper[2573]: W0709 15:31:17.411433 2573 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 09 15:31:17.411875 ip-10-0-133-183 kubenswrapper[2573]: W0709 15:31:17.411436 2573 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 09 15:31:17.411875 ip-10-0-133-183 kubenswrapper[2573]: W0709 15:31:17.411439 2573 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 09 15:31:17.411875 ip-10-0-133-183 kubenswrapper[2573]: W0709 15:31:17.411442 2573 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 09 15:31:17.411875 ip-10-0-133-183 kubenswrapper[2573]: W0709 15:31:17.411445 2573 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 09 15:31:17.411875 ip-10-0-133-183 kubenswrapper[2573]: W0709 15:31:17.411448 2573 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 09 15:31:17.411875 ip-10-0-133-183 kubenswrapper[2573]: W0709 15:31:17.411451 2573 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 09 15:31:17.411875 ip-10-0-133-183 kubenswrapper[2573]: W0709 15:31:17.411453 2573 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 09 15:31:17.411875 ip-10-0-133-183 kubenswrapper[2573]: W0709 15:31:17.411456 2573 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 09 15:31:17.411875 ip-10-0-133-183 kubenswrapper[2573]: W0709 15:31:17.411459 2573 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 09 15:31:17.411875 ip-10-0-133-183 kubenswrapper[2573]: W0709 15:31:17.411461 2573 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 09 15:31:17.411875 ip-10-0-133-183 kubenswrapper[2573]: W0709 15:31:17.411464 2573 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 09 15:31:17.411875 ip-10-0-133-183 kubenswrapper[2573]: W0709 15:31:17.411466 2573 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 09 15:31:17.411875 ip-10-0-133-183 kubenswrapper[2573]: W0709 15:31:17.411469 2573 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 09 15:31:17.411875 ip-10-0-133-183 kubenswrapper[2573]: W0709 15:31:17.411471 2573 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 09 15:31:17.411875 ip-10-0-133-183 kubenswrapper[2573]: W0709 15:31:17.411474 2573 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 09 15:31:17.411875 ip-10-0-133-183 kubenswrapper[2573]: W0709 15:31:17.411476 2573 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 09 15:31:17.411875 ip-10-0-133-183 kubenswrapper[2573]: W0709 15:31:17.411479 2573 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 09 15:31:17.411875 ip-10-0-133-183 kubenswrapper[2573]: W0709 15:31:17.411481 2573 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 09 15:31:17.411875 ip-10-0-133-183 kubenswrapper[2573]: W0709 15:31:17.411484 2573 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 09 15:31:17.412353 ip-10-0-133-183 kubenswrapper[2573]: W0709 15:31:17.411486 2573 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 09 15:31:17.412353 ip-10-0-133-183 kubenswrapper[2573]: W0709 15:31:17.411489 2573 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 09 15:31:17.412353 ip-10-0-133-183 kubenswrapper[2573]: W0709 15:31:17.411491 2573 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 09 15:31:17.412353 ip-10-0-133-183 kubenswrapper[2573]: W0709 15:31:17.411494 2573 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 09 15:31:17.412353 ip-10-0-133-183 kubenswrapper[2573]: W0709 15:31:17.411496 2573 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 09 15:31:17.412353 ip-10-0-133-183 kubenswrapper[2573]: W0709 15:31:17.411499 2573 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 09 15:31:17.412353 ip-10-0-133-183 kubenswrapper[2573]: W0709 15:31:17.411501 2573 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 09 15:31:17.412353 ip-10-0-133-183 kubenswrapper[2573]: W0709 15:31:17.411503 2573 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 09 15:31:17.412353 ip-10-0-133-183 kubenswrapper[2573]: W0709 15:31:17.411506 2573 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 09 15:31:17.412353 ip-10-0-133-183 kubenswrapper[2573]: W0709 15:31:17.411508 2573 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 09 15:31:17.412353 ip-10-0-133-183 kubenswrapper[2573]: W0709 15:31:17.411511 2573 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 09 15:31:17.412353 ip-10-0-133-183 kubenswrapper[2573]: W0709 15:31:17.411513 2573 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 09 15:31:17.412353 ip-10-0-133-183 kubenswrapper[2573]: W0709 15:31:17.411515 2573 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 09 15:31:17.412353 ip-10-0-133-183 kubenswrapper[2573]: W0709 15:31:17.411519 2573 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 09 15:31:17.412353 ip-10-0-133-183 kubenswrapper[2573]: W0709 15:31:17.411521 2573 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 09 15:31:17.412353 ip-10-0-133-183 kubenswrapper[2573]: W0709 15:31:17.411524 2573 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 09 15:31:17.412353 ip-10-0-133-183 kubenswrapper[2573]: W0709 15:31:17.411527 2573 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 09 15:31:17.412353 ip-10-0-133-183 kubenswrapper[2573]: W0709 15:31:17.411529 2573 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 09 15:31:17.412353 ip-10-0-133-183 kubenswrapper[2573]: W0709 15:31:17.411531 2573 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 09 15:31:17.412818 ip-10-0-133-183 kubenswrapper[2573]: W0709 15:31:17.411534 2573 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 09 15:31:17.412818 ip-10-0-133-183 kubenswrapper[2573]: W0709 15:31:17.411536 2573 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 09 15:31:17.412818 ip-10-0-133-183 kubenswrapper[2573]: W0709 15:31:17.411539 2573 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 09 15:31:17.412818 ip-10-0-133-183 kubenswrapper[2573]: W0709 15:31:17.411542 2573 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 09 15:31:17.412818 ip-10-0-133-183 kubenswrapper[2573]: W0709 15:31:17.411544 2573 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 09 15:31:17.412818 ip-10-0-133-183 kubenswrapper[2573]: W0709 15:31:17.411547 2573 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 09 15:31:17.412818 ip-10-0-133-183 kubenswrapper[2573]: W0709 15:31:17.411549 2573 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 09 15:31:17.412818 ip-10-0-133-183 kubenswrapper[2573]: W0709 15:31:17.411552 2573 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 09 15:31:17.412818 ip-10-0-133-183 kubenswrapper[2573]: W0709 15:31:17.411555 2573 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 09 15:31:17.412818 ip-10-0-133-183 kubenswrapper[2573]: W0709 15:31:17.411557 2573 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 09 15:31:17.412818 ip-10-0-133-183 kubenswrapper[2573]: W0709 15:31:17.411560 2573 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 09 15:31:17.412818 ip-10-0-133-183 kubenswrapper[2573]: W0709 15:31:17.411563 2573 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 09 15:31:17.412818 ip-10-0-133-183 kubenswrapper[2573]: W0709 15:31:17.411566 2573 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 09 15:31:17.412818 ip-10-0-133-183 kubenswrapper[2573]: W0709 15:31:17.411568 2573 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 09 15:31:17.412818 ip-10-0-133-183 kubenswrapper[2573]: W0709 15:31:17.411571 2573 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 09 15:31:17.412818 ip-10-0-133-183 kubenswrapper[2573]: W0709 15:31:17.411573 2573 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 09 15:31:17.412818 ip-10-0-133-183 kubenswrapper[2573]: W0709 15:31:17.411576 2573 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 09 15:31:17.412818 ip-10-0-133-183 kubenswrapper[2573]: W0709 15:31:17.411579 2573 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 09 15:31:17.412818 ip-10-0-133-183 kubenswrapper[2573]: W0709 15:31:17.411582 2573 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 09 15:31:17.412818 ip-10-0-133-183 kubenswrapper[2573]: W0709 15:31:17.411586 2573 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 09 15:31:17.413318 ip-10-0-133-183 kubenswrapper[2573]: W0709 15:31:17.411590 2573 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 09 15:31:17.413318 ip-10-0-133-183 kubenswrapper[2573]: W0709 15:31:17.411593 2573 feature_gate.go:328] unrecognized feature gate: Example Jul 09 15:31:17.413318 ip-10-0-133-183 kubenswrapper[2573]: W0709 15:31:17.411596 2573 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 09 15:31:17.413318 ip-10-0-133-183 kubenswrapper[2573]: W0709 15:31:17.411598 2573 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 09 15:31:17.413318 ip-10-0-133-183 kubenswrapper[2573]: W0709 15:31:17.411601 2573 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 09 15:31:17.413318 ip-10-0-133-183 kubenswrapper[2573]: W0709 15:31:17.411604 2573 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 09 15:31:17.413318 ip-10-0-133-183 kubenswrapper[2573]: W0709 15:31:17.411607 2573 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 09 15:31:17.413318 ip-10-0-133-183 kubenswrapper[2573]: W0709 15:31:17.411610 2573 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 09 15:31:17.413318 ip-10-0-133-183 kubenswrapper[2573]: W0709 15:31:17.411613 2573 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 09 15:31:17.413318 ip-10-0-133-183 kubenswrapper[2573]: W0709 15:31:17.411615 2573 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 09 15:31:17.413318 ip-10-0-133-183 kubenswrapper[2573]: W0709 15:31:17.411618 2573 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 09 15:31:17.413318 ip-10-0-133-183 kubenswrapper[2573]: W0709 15:31:17.411620 2573 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 09 15:31:17.413318 ip-10-0-133-183 kubenswrapper[2573]: W0709 15:31:17.411623 2573 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 09 15:31:17.413318 ip-10-0-133-183 kubenswrapper[2573]: W0709 15:31:17.411625 2573 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 09 15:31:17.413318 ip-10-0-133-183 kubenswrapper[2573]: W0709 15:31:17.411628 2573 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 09 15:31:17.413318 ip-10-0-133-183 kubenswrapper[2573]: W0709 15:31:17.411630 2573 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 09 15:31:17.413318 ip-10-0-133-183 kubenswrapper[2573]: W0709 15:31:17.411633 2573 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 09 15:31:17.413318 ip-10-0-133-183 kubenswrapper[2573]: W0709 15:31:17.411636 2573 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 09 15:31:17.413318 ip-10-0-133-183 kubenswrapper[2573]: W0709 15:31:17.411638 2573 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 09 15:31:17.413318 ip-10-0-133-183 kubenswrapper[2573]: W0709 15:31:17.412377 2573 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 09 15:31:17.413798 ip-10-0-133-183 kubenswrapper[2573]: W0709 15:31:17.412384 2573 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 09 15:31:17.413798 ip-10-0-133-183 kubenswrapper[2573]: W0709 15:31:17.412387 2573 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 09 15:31:17.413798 ip-10-0-133-183 kubenswrapper[2573]: W0709 15:31:17.412389 2573 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 09 15:31:17.413798 ip-10-0-133-183 kubenswrapper[2573]: W0709 15:31:17.412392 2573 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 09 15:31:17.413798 ip-10-0-133-183 kubenswrapper[2573]: W0709 15:31:17.412395 2573 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 09 15:31:17.413798 ip-10-0-133-183 kubenswrapper[2573]: W0709 15:31:17.412398 2573 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 09 15:31:17.413798 ip-10-0-133-183 kubenswrapper[2573]: W0709 15:31:17.412402 2573 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 09 15:31:17.413798 ip-10-0-133-183 kubenswrapper[2573]: W0709 15:31:17.412405 2573 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 09 15:31:17.413798 ip-10-0-133-183 kubenswrapper[2573]: W0709 15:31:17.412407 2573 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 09 15:31:17.413798 ip-10-0-133-183 kubenswrapper[2573]: W0709 15:31:17.412410 2573 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 09 15:31:17.413798 ip-10-0-133-183 kubenswrapper[2573]: W0709 15:31:17.412413 2573 feature_gate.go:328] unrecognized feature gate: Example Jul 09 15:31:17.413798 ip-10-0-133-183 kubenswrapper[2573]: W0709 15:31:17.412415 2573 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 09 15:31:17.413798 ip-10-0-133-183 kubenswrapper[2573]: W0709 15:31:17.412418 2573 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 09 15:31:17.413798 ip-10-0-133-183 kubenswrapper[2573]: W0709 15:31:17.412420 2573 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 09 15:31:17.413798 ip-10-0-133-183 kubenswrapper[2573]: W0709 15:31:17.412423 2573 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 09 15:31:17.413798 ip-10-0-133-183 kubenswrapper[2573]: W0709 15:31:17.412426 2573 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 09 15:31:17.413798 ip-10-0-133-183 kubenswrapper[2573]: W0709 15:31:17.412428 2573 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 09 15:31:17.413798 ip-10-0-133-183 kubenswrapper[2573]: W0709 15:31:17.412431 2573 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 09 15:31:17.413798 ip-10-0-133-183 kubenswrapper[2573]: W0709 15:31:17.412435 2573 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 09 15:31:17.413798 ip-10-0-133-183 kubenswrapper[2573]: W0709 15:31:17.412438 2573 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 09 15:31:17.414303 ip-10-0-133-183 kubenswrapper[2573]: W0709 15:31:17.412440 2573 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 09 15:31:17.414303 ip-10-0-133-183 kubenswrapper[2573]: W0709 15:31:17.412443 2573 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 09 15:31:17.414303 ip-10-0-133-183 kubenswrapper[2573]: W0709 15:31:17.412446 2573 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 09 15:31:17.414303 ip-10-0-133-183 kubenswrapper[2573]: W0709 15:31:17.412449 2573 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 09 15:31:17.414303 ip-10-0-133-183 kubenswrapper[2573]: W0709 15:31:17.412451 2573 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 09 15:31:17.414303 ip-10-0-133-183 kubenswrapper[2573]: W0709 15:31:17.412454 2573 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 09 15:31:17.414303 ip-10-0-133-183 kubenswrapper[2573]: W0709 15:31:17.412456 2573 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 09 15:31:17.414303 ip-10-0-133-183 kubenswrapper[2573]: W0709 15:31:17.412459 2573 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 09 15:31:17.414303 ip-10-0-133-183 kubenswrapper[2573]: W0709 15:31:17.412462 2573 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 09 15:31:17.414303 ip-10-0-133-183 kubenswrapper[2573]: W0709 15:31:17.412464 2573 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 09 15:31:17.414303 ip-10-0-133-183 kubenswrapper[2573]: W0709 15:31:17.412467 2573 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 09 15:31:17.414303 ip-10-0-133-183 kubenswrapper[2573]: W0709 15:31:17.412470 2573 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 09 15:31:17.414303 ip-10-0-133-183 kubenswrapper[2573]: W0709 15:31:17.412472 2573 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 09 15:31:17.414303 ip-10-0-133-183 kubenswrapper[2573]: W0709 15:31:17.412475 2573 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 09 15:31:17.414303 ip-10-0-133-183 kubenswrapper[2573]: W0709 15:31:17.412477 2573 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 09 15:31:17.414303 ip-10-0-133-183 kubenswrapper[2573]: W0709 15:31:17.412479 2573 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 09 15:31:17.414303 ip-10-0-133-183 kubenswrapper[2573]: W0709 15:31:17.412482 2573 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 09 15:31:17.414303 ip-10-0-133-183 kubenswrapper[2573]: W0709 15:31:17.412485 2573 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 09 15:31:17.414303 ip-10-0-133-183 kubenswrapper[2573]: W0709 15:31:17.412487 2573 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 09 15:31:17.414763 ip-10-0-133-183 kubenswrapper[2573]: W0709 15:31:17.412490 2573 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 09 15:31:17.414763 ip-10-0-133-183 kubenswrapper[2573]: W0709 15:31:17.412493 2573 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 09 15:31:17.414763 ip-10-0-133-183 kubenswrapper[2573]: W0709 15:31:17.412496 2573 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 09 15:31:17.414763 ip-10-0-133-183 kubenswrapper[2573]: W0709 15:31:17.412498 2573 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 09 15:31:17.414763 ip-10-0-133-183 kubenswrapper[2573]: W0709 15:31:17.412501 2573 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 09 15:31:17.414763 ip-10-0-133-183 kubenswrapper[2573]: W0709 15:31:17.412503 2573 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 09 15:31:17.414763 ip-10-0-133-183 kubenswrapper[2573]: W0709 15:31:17.412506 2573 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 09 15:31:17.414763 ip-10-0-133-183 kubenswrapper[2573]: W0709 15:31:17.412509 2573 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 09 15:31:17.414763 ip-10-0-133-183 kubenswrapper[2573]: W0709 15:31:17.412511 2573 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 09 15:31:17.414763 ip-10-0-133-183 kubenswrapper[2573]: W0709 15:31:17.412514 2573 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 09 15:31:17.414763 ip-10-0-133-183 kubenswrapper[2573]: W0709 15:31:17.412516 2573 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 09 15:31:17.414763 ip-10-0-133-183 kubenswrapper[2573]: W0709 15:31:17.412519 2573 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 09 15:31:17.414763 ip-10-0-133-183 kubenswrapper[2573]: W0709 15:31:17.412522 2573 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 09 15:31:17.414763 ip-10-0-133-183 kubenswrapper[2573]: W0709 15:31:17.412524 2573 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 09 15:31:17.414763 ip-10-0-133-183 kubenswrapper[2573]: W0709 15:31:17.412527 2573 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 09 15:31:17.414763 ip-10-0-133-183 kubenswrapper[2573]: W0709 15:31:17.412530 2573 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 09 15:31:17.414763 ip-10-0-133-183 kubenswrapper[2573]: W0709 15:31:17.412532 2573 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 09 15:31:17.414763 ip-10-0-133-183 kubenswrapper[2573]: W0709 15:31:17.412538 2573 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 09 15:31:17.414763 ip-10-0-133-183 kubenswrapper[2573]: W0709 15:31:17.412541 2573 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 09 15:31:17.415247 ip-10-0-133-183 kubenswrapper[2573]: W0709 15:31:17.412544 2573 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 09 15:31:17.415247 ip-10-0-133-183 kubenswrapper[2573]: W0709 15:31:17.412547 2573 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 09 15:31:17.415247 ip-10-0-133-183 kubenswrapper[2573]: W0709 15:31:17.412550 2573 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 09 15:31:17.415247 ip-10-0-133-183 kubenswrapper[2573]: W0709 15:31:17.412553 2573 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 09 15:31:17.415247 ip-10-0-133-183 kubenswrapper[2573]: W0709 15:31:17.412565 2573 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 09 15:31:17.415247 ip-10-0-133-183 kubenswrapper[2573]: W0709 15:31:17.412570 2573 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 09 15:31:17.415247 ip-10-0-133-183 kubenswrapper[2573]: W0709 15:31:17.412574 2573 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 09 15:31:17.415247 ip-10-0-133-183 kubenswrapper[2573]: W0709 15:31:17.412577 2573 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 09 15:31:17.415247 ip-10-0-133-183 kubenswrapper[2573]: W0709 15:31:17.412580 2573 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 09 15:31:17.415247 ip-10-0-133-183 kubenswrapper[2573]: W0709 15:31:17.412583 2573 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 09 15:31:17.415247 ip-10-0-133-183 kubenswrapper[2573]: W0709 15:31:17.412586 2573 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 09 15:31:17.415247 ip-10-0-133-183 kubenswrapper[2573]: W0709 15:31:17.412589 2573 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 09 15:31:17.415247 ip-10-0-133-183 kubenswrapper[2573]: W0709 15:31:17.412593 2573 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 09 15:31:17.415247 ip-10-0-133-183 kubenswrapper[2573]: W0709 15:31:17.412596 2573 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 09 15:31:17.415247 ip-10-0-133-183 kubenswrapper[2573]: W0709 15:31:17.412599 2573 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 09 15:31:17.415247 ip-10-0-133-183 kubenswrapper[2573]: W0709 15:31:17.412602 2573 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 09 15:31:17.415247 ip-10-0-133-183 kubenswrapper[2573]: W0709 15:31:17.412604 2573 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 09 15:31:17.415247 ip-10-0-133-183 kubenswrapper[2573]: W0709 15:31:17.412607 2573 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 09 15:31:17.415247 ip-10-0-133-183 kubenswrapper[2573]: W0709 15:31:17.412610 2573 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 09 15:31:17.415247 ip-10-0-133-183 kubenswrapper[2573]: W0709 15:31:17.412613 2573 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 09 15:31:17.415727 ip-10-0-133-183 kubenswrapper[2573]: W0709 15:31:17.412618 2573 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 09 15:31:17.415727 ip-10-0-133-183 kubenswrapper[2573]: W0709 15:31:17.412621 2573 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 09 15:31:17.415727 ip-10-0-133-183 kubenswrapper[2573]: W0709 15:31:17.412624 2573 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 09 15:31:17.415727 ip-10-0-133-183 kubenswrapper[2573]: W0709 15:31:17.412627 2573 feature_gate.go:328] unrecognized feature gate: Example2 Jul 09 15:31:17.415727 ip-10-0-133-183 kubenswrapper[2573]: W0709 15:31:17.412630 2573 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 09 15:31:17.415727 ip-10-0-133-183 kubenswrapper[2573]: W0709 15:31:17.412633 2573 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 09 15:31:17.415727 ip-10-0-133-183 kubenswrapper[2573]: W0709 15:31:17.412635 2573 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 09 15:31:17.415727 ip-10-0-133-183 kubenswrapper[2573]: W0709 15:31:17.412638 2573 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 09 15:31:17.415727 ip-10-0-133-183 kubenswrapper[2573]: W0709 15:31:17.412640 2573 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 09 15:31:17.415727 ip-10-0-133-183 kubenswrapper[2573]: W0709 15:31:17.412643 2573 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 09 15:31:17.415727 ip-10-0-133-183 kubenswrapper[2573]: W0709 15:31:17.412646 2573 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 09 15:31:17.415727 ip-10-0-133-183 kubenswrapper[2573]: W0709 15:31:17.412649 2573 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 09 15:31:17.415727 ip-10-0-133-183 kubenswrapper[2573]: W0709 15:31:17.412651 2573 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 09 15:31:17.415727 ip-10-0-133-183 kubenswrapper[2573]: W0709 15:31:17.412654 2573 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 09 15:31:17.415727 ip-10-0-133-183 kubenswrapper[2573]: W0709 15:31:17.412656 2573 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 09 15:31:17.415727 ip-10-0-133-183 kubenswrapper[2573]: W0709 15:31:17.412659 2573 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 09 15:31:17.415727 ip-10-0-133-183 kubenswrapper[2573]: W0709 15:31:17.412662 2573 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 09 15:31:17.415727 ip-10-0-133-183 kubenswrapper[2573]: W0709 15:31:17.412664 2573 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 09 15:31:17.415727 ip-10-0-133-183 kubenswrapper[2573]: W0709 15:31:17.412667 2573 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 09 15:31:17.415727 ip-10-0-133-183 kubenswrapper[2573]: W0709 15:31:17.412669 2573 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 09 15:31:17.416253 ip-10-0-133-183 kubenswrapper[2573]: W0709 15:31:17.413070 2573 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 09 15:31:17.416253 ip-10-0-133-183 kubenswrapper[2573]: W0709 15:31:17.413076 2573 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 09 15:31:17.416253 ip-10-0-133-183 kubenswrapper[2573]: W0709 15:31:17.413078 2573 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 09 15:31:17.416253 ip-10-0-133-183 kubenswrapper[2573]: W0709 15:31:17.413081 2573 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 09 15:31:17.416253 ip-10-0-133-183 kubenswrapper[2573]: W0709 15:31:17.413084 2573 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 09 15:31:17.416253 ip-10-0-133-183 kubenswrapper[2573]: W0709 15:31:17.413087 2573 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 09 15:31:17.416253 ip-10-0-133-183 kubenswrapper[2573]: W0709 15:31:17.413090 2573 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 09 15:31:17.416253 ip-10-0-133-183 kubenswrapper[2573]: W0709 15:31:17.413093 2573 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 09 15:31:17.416253 ip-10-0-133-183 kubenswrapper[2573]: W0709 15:31:17.413096 2573 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 09 15:31:17.416253 ip-10-0-133-183 kubenswrapper[2573]: W0709 15:31:17.413099 2573 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 09 15:31:17.416253 ip-10-0-133-183 kubenswrapper[2573]: W0709 15:31:17.413101 2573 feature_gate.go:328] unrecognized feature gate: Example Jul 09 15:31:17.416253 ip-10-0-133-183 kubenswrapper[2573]: W0709 15:31:17.413104 2573 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 09 15:31:17.416253 ip-10-0-133-183 kubenswrapper[2573]: W0709 15:31:17.413107 2573 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 09 15:31:17.416253 ip-10-0-133-183 kubenswrapper[2573]: W0709 15:31:17.413109 2573 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 09 15:31:17.416253 ip-10-0-133-183 kubenswrapper[2573]: W0709 15:31:17.413112 2573 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 09 15:31:17.416253 ip-10-0-133-183 kubenswrapper[2573]: W0709 15:31:17.413114 2573 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 09 15:31:17.416253 ip-10-0-133-183 kubenswrapper[2573]: W0709 15:31:17.413117 2573 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 09 15:31:17.416253 ip-10-0-133-183 kubenswrapper[2573]: W0709 15:31:17.413120 2573 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 09 15:31:17.416253 ip-10-0-133-183 kubenswrapper[2573]: W0709 15:31:17.413123 2573 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 09 15:31:17.416253 ip-10-0-133-183 kubenswrapper[2573]: W0709 15:31:17.413125 2573 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 09 15:31:17.416749 ip-10-0-133-183 kubenswrapper[2573]: W0709 15:31:17.413128 2573 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 09 15:31:17.416749 ip-10-0-133-183 kubenswrapper[2573]: W0709 15:31:17.413131 2573 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 09 15:31:17.416749 ip-10-0-133-183 kubenswrapper[2573]: W0709 15:31:17.413133 2573 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 09 15:31:17.416749 ip-10-0-133-183 kubenswrapper[2573]: W0709 15:31:17.413136 2573 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 09 15:31:17.416749 ip-10-0-133-183 kubenswrapper[2573]: W0709 15:31:17.413139 2573 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 09 15:31:17.416749 ip-10-0-133-183 kubenswrapper[2573]: W0709 15:31:17.413141 2573 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 09 15:31:17.416749 ip-10-0-133-183 kubenswrapper[2573]: W0709 15:31:17.413144 2573 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 09 15:31:17.416749 ip-10-0-133-183 kubenswrapper[2573]: W0709 15:31:17.413147 2573 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 09 15:31:17.416749 ip-10-0-133-183 kubenswrapper[2573]: W0709 15:31:17.413149 2573 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 09 15:31:17.416749 ip-10-0-133-183 kubenswrapper[2573]: W0709 15:31:17.413152 2573 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 09 15:31:17.416749 ip-10-0-133-183 kubenswrapper[2573]: W0709 15:31:17.413154 2573 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 09 15:31:17.416749 ip-10-0-133-183 kubenswrapper[2573]: W0709 15:31:17.413157 2573 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 09 15:31:17.416749 ip-10-0-133-183 kubenswrapper[2573]: W0709 15:31:17.413160 2573 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 09 15:31:17.416749 ip-10-0-133-183 kubenswrapper[2573]: W0709 15:31:17.413162 2573 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 09 15:31:17.416749 ip-10-0-133-183 kubenswrapper[2573]: W0709 15:31:17.413165 2573 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 09 15:31:17.416749 ip-10-0-133-183 kubenswrapper[2573]: W0709 15:31:17.413167 2573 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 09 15:31:17.416749 ip-10-0-133-183 kubenswrapper[2573]: W0709 15:31:17.413170 2573 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 09 15:31:17.416749 ip-10-0-133-183 kubenswrapper[2573]: W0709 15:31:17.413173 2573 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 09 15:31:17.416749 ip-10-0-133-183 kubenswrapper[2573]: W0709 15:31:17.413176 2573 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 09 15:31:17.416749 ip-10-0-133-183 kubenswrapper[2573]: W0709 15:31:17.413178 2573 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 09 15:31:17.417273 ip-10-0-133-183 kubenswrapper[2573]: W0709 15:31:17.413181 2573 feature_gate.go:328] unrecognized feature gate: Example2 Jul 09 15:31:17.417273 ip-10-0-133-183 kubenswrapper[2573]: W0709 15:31:17.413184 2573 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 09 15:31:17.417273 ip-10-0-133-183 kubenswrapper[2573]: W0709 15:31:17.413186 2573 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 09 15:31:17.417273 ip-10-0-133-183 kubenswrapper[2573]: W0709 15:31:17.413189 2573 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 09 15:31:17.417273 ip-10-0-133-183 kubenswrapper[2573]: W0709 15:31:17.413191 2573 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 09 15:31:17.417273 ip-10-0-133-183 kubenswrapper[2573]: W0709 15:31:17.413194 2573 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 09 15:31:17.417273 ip-10-0-133-183 kubenswrapper[2573]: W0709 15:31:17.413197 2573 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 09 15:31:17.417273 ip-10-0-133-183 kubenswrapper[2573]: W0709 15:31:17.413199 2573 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 09 15:31:17.417273 ip-10-0-133-183 kubenswrapper[2573]: W0709 15:31:17.413203 2573 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 09 15:31:17.417273 ip-10-0-133-183 kubenswrapper[2573]: W0709 15:31:17.413206 2573 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 09 15:31:17.417273 ip-10-0-133-183 kubenswrapper[2573]: W0709 15:31:17.413209 2573 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 09 15:31:17.417273 ip-10-0-133-183 kubenswrapper[2573]: W0709 15:31:17.413211 2573 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 09 15:31:17.417273 ip-10-0-133-183 kubenswrapper[2573]: W0709 15:31:17.413214 2573 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 09 15:31:17.417273 ip-10-0-133-183 kubenswrapper[2573]: W0709 15:31:17.413216 2573 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 09 15:31:17.417273 ip-10-0-133-183 kubenswrapper[2573]: W0709 15:31:17.413219 2573 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 09 15:31:17.417273 ip-10-0-133-183 kubenswrapper[2573]: W0709 15:31:17.413222 2573 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 09 15:31:17.417273 ip-10-0-133-183 kubenswrapper[2573]: W0709 15:31:17.413225 2573 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 09 15:31:17.417273 ip-10-0-133-183 kubenswrapper[2573]: W0709 15:31:17.413228 2573 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 09 15:31:17.417273 ip-10-0-133-183 kubenswrapper[2573]: W0709 15:31:17.413231 2573 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 09 15:31:17.417722 ip-10-0-133-183 kubenswrapper[2573]: W0709 15:31:17.413235 2573 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 09 15:31:17.417722 ip-10-0-133-183 kubenswrapper[2573]: W0709 15:31:17.413239 2573 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 09 15:31:17.417722 ip-10-0-133-183 kubenswrapper[2573]: W0709 15:31:17.413242 2573 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 09 15:31:17.417722 ip-10-0-133-183 kubenswrapper[2573]: W0709 15:31:17.413245 2573 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 09 15:31:17.417722 ip-10-0-133-183 kubenswrapper[2573]: W0709 15:31:17.413248 2573 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 09 15:31:17.417722 ip-10-0-133-183 kubenswrapper[2573]: W0709 15:31:17.413251 2573 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 09 15:31:17.417722 ip-10-0-133-183 kubenswrapper[2573]: W0709 15:31:17.413254 2573 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 09 15:31:17.417722 ip-10-0-133-183 kubenswrapper[2573]: W0709 15:31:17.413256 2573 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 09 15:31:17.417722 ip-10-0-133-183 kubenswrapper[2573]: W0709 15:31:17.413260 2573 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 09 15:31:17.417722 ip-10-0-133-183 kubenswrapper[2573]: W0709 15:31:17.413263 2573 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 09 15:31:17.417722 ip-10-0-133-183 kubenswrapper[2573]: W0709 15:31:17.413268 2573 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 09 15:31:17.417722 ip-10-0-133-183 kubenswrapper[2573]: W0709 15:31:17.413275 2573 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 09 15:31:17.417722 ip-10-0-133-183 kubenswrapper[2573]: W0709 15:31:17.413278 2573 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 09 15:31:17.417722 ip-10-0-133-183 kubenswrapper[2573]: W0709 15:31:17.413281 2573 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 09 15:31:17.417722 ip-10-0-133-183 kubenswrapper[2573]: W0709 15:31:17.413284 2573 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 09 15:31:17.417722 ip-10-0-133-183 kubenswrapper[2573]: W0709 15:31:17.413287 2573 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 09 15:31:17.417722 ip-10-0-133-183 kubenswrapper[2573]: W0709 15:31:17.413290 2573 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 09 15:31:17.417722 ip-10-0-133-183 kubenswrapper[2573]: W0709 15:31:17.413293 2573 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 09 15:31:17.417722 ip-10-0-133-183 kubenswrapper[2573]: W0709 15:31:17.413295 2573 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 09 15:31:17.417722 ip-10-0-133-183 kubenswrapper[2573]: W0709 15:31:17.413298 2573 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 09 15:31:17.418207 ip-10-0-133-183 kubenswrapper[2573]: W0709 15:31:17.413301 2573 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 09 15:31:17.418207 ip-10-0-133-183 kubenswrapper[2573]: W0709 15:31:17.413303 2573 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 09 15:31:17.418207 ip-10-0-133-183 kubenswrapper[2573]: W0709 15:31:17.413306 2573 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 09 15:31:17.418207 ip-10-0-133-183 kubenswrapper[2573]: W0709 15:31:17.413308 2573 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 09 15:31:17.418207 ip-10-0-133-183 kubenswrapper[2573]: W0709 15:31:17.413311 2573 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 09 15:31:17.418207 ip-10-0-133-183 kubenswrapper[2573]: W0709 15:31:17.413313 2573 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 09 15:31:17.418207 ip-10-0-133-183 kubenswrapper[2573]: W0709 15:31:17.413316 2573 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 09 15:31:17.418207 ip-10-0-133-183 kubenswrapper[2573]: W0709 15:31:17.413318 2573 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 09 15:31:17.418207 ip-10-0-133-183 kubenswrapper[2573]: W0709 15:31:17.413321 2573 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 09 15:31:17.418207 ip-10-0-133-183 kubenswrapper[2573]: W0709 15:31:17.413323 2573 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 09 15:31:17.418207 ip-10-0-133-183 kubenswrapper[2573]: W0709 15:31:17.413326 2573 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 09 15:31:17.418207 ip-10-0-133-183 kubenswrapper[2573]: W0709 15:31:17.413329 2573 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 09 15:31:17.418207 ip-10-0-133-183 kubenswrapper[2573]: W0709 15:31:17.413331 2573 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 09 15:31:17.418207 ip-10-0-133-183 kubenswrapper[2573]: W0709 15:31:17.413334 2573 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 09 15:31:17.418207 ip-10-0-133-183 kubenswrapper[2573]: W0709 15:31:17.413336 2573 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 09 15:31:17.418207 ip-10-0-133-183 kubenswrapper[2573]: W0709 15:31:17.413339 2573 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 09 15:31:17.418207 ip-10-0-133-183 kubenswrapper[2573]: W0709 15:31:17.413342 2573 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 09 15:31:17.418207 ip-10-0-133-183 kubenswrapper[2573]: W0709 15:31:17.413344 2573 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 09 15:31:17.418207 ip-10-0-133-183 kubenswrapper[2573]: W0709 15:31:17.413347 2573 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 09 15:31:17.418207 ip-10-0-133-183 kubenswrapper[2573]: W0709 15:31:17.413350 2573 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 09 15:31:17.418722 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:31:17.413427 2573 flags.go:64] FLAG: --address="0.0.0.0" Jul 09 15:31:17.418722 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:31:17.413439 2573 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]" Jul 09 15:31:17.418722 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:31:17.413448 2573 flags.go:64] FLAG: --anonymous-auth="true" Jul 09 15:31:17.418722 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:31:17.413460 2573 flags.go:64] FLAG: --application-metrics-count-limit="100" Jul 09 15:31:17.418722 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:31:17.413468 2573 flags.go:64] FLAG: --authentication-token-webhook="false" Jul 09 15:31:17.418722 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:31:17.413473 2573 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s" Jul 09 15:31:17.418722 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:31:17.413477 2573 flags.go:64] FLAG: --authorization-mode="AlwaysAllow" Jul 09 15:31:17.418722 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:31:17.413482 2573 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s" Jul 09 15:31:17.418722 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:31:17.413486 2573 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s" Jul 09 15:31:17.418722 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:31:17.413489 2573 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id" Jul 09 15:31:17.418722 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:31:17.413492 2573 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig" Jul 09 15:31:17.418722 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:31:17.413496 2573 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki" Jul 09 15:31:17.418722 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:31:17.413499 2573 flags.go:64] FLAG: --cgroup-driver="cgroupfs" Jul 09 15:31:17.418722 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:31:17.413502 2573 flags.go:64] FLAG: --cgroup-root="" Jul 09 15:31:17.418722 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:31:17.413505 2573 flags.go:64] FLAG: --cgroups-per-qos="true" Jul 09 15:31:17.418722 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:31:17.413508 2573 flags.go:64] FLAG: --client-ca-file="" Jul 09 15:31:17.418722 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:31:17.413511 2573 flags.go:64] FLAG: --cloud-provider="external" Jul 09 15:31:17.418722 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:31:17.413514 2573 flags.go:64] FLAG: --cluster-dns="[]" Jul 09 15:31:17.418722 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:31:17.413520 2573 flags.go:64] FLAG: --cluster-domain="" Jul 09 15:31:17.418722 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:31:17.413523 2573 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf" Jul 09 15:31:17.418722 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:31:17.413526 2573 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d" Jul 09 15:31:17.418722 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:31:17.413529 2573 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json" Jul 09 15:31:17.418722 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:31:17.413532 2573 flags.go:64] FLAG: --container-log-max-files="5" Jul 09 15:31:17.418722 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:31:17.413536 2573 flags.go:64] FLAG: --container-log-max-size="10Mi" Jul 09 15:31:17.419303 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:31:17.413539 2573 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock" Jul 09 15:31:17.419303 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:31:17.413542 2573 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock" Jul 09 15:31:17.419303 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:31:17.413545 2573 flags.go:64] FLAG: --containerd-namespace="k8s.io" Jul 09 15:31:17.419303 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:31:17.413548 2573 flags.go:64] FLAG: --contention-profiling="false" Jul 09 15:31:17.419303 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:31:17.413551 2573 flags.go:64] FLAG: --cpu-cfs-quota="true" Jul 09 15:31:17.419303 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:31:17.413554 2573 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms" Jul 09 15:31:17.419303 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:31:17.413557 2573 flags.go:64] FLAG: --cpu-manager-policy="none" Jul 09 15:31:17.419303 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:31:17.413560 2573 flags.go:64] FLAG: --cpu-manager-policy-options="" Jul 09 15:31:17.419303 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:31:17.413564 2573 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s" Jul 09 15:31:17.419303 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:31:17.413568 2573 flags.go:64] FLAG: --enable-controller-attach-detach="true" Jul 09 15:31:17.419303 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:31:17.413570 2573 flags.go:64] FLAG: --enable-debugging-handlers="true" Jul 09 15:31:17.419303 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:31:17.413575 2573 flags.go:64] FLAG: --enable-load-reader="false" Jul 09 15:31:17.419303 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:31:17.413578 2573 flags.go:64] FLAG: --enable-server="true" Jul 09 15:31:17.419303 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:31:17.413582 2573 flags.go:64] FLAG: --enforce-node-allocatable="[pods]" Jul 09 15:31:17.419303 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:31:17.413586 2573 flags.go:64] FLAG: --event-burst="100" Jul 09 15:31:17.419303 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:31:17.413590 2573 flags.go:64] FLAG: --event-qps="50" Jul 09 15:31:17.419303 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:31:17.413592 2573 flags.go:64] FLAG: --event-storage-age-limit="default=0" Jul 09 15:31:17.419303 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:31:17.413596 2573 flags.go:64] FLAG: --event-storage-event-limit="default=0" Jul 09 15:31:17.419303 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:31:17.413599 2573 flags.go:64] FLAG: --eviction-hard="" Jul 09 15:31:17.419303 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:31:17.413602 2573 flags.go:64] FLAG: --eviction-max-pod-grace-period="0" Jul 09 15:31:17.419303 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:31:17.413605 2573 flags.go:64] FLAG: --eviction-minimum-reclaim="" Jul 09 15:31:17.419303 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:31:17.413608 2573 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s" Jul 09 15:31:17.419303 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:31:17.413611 2573 flags.go:64] FLAG: --eviction-soft="" Jul 09 15:31:17.419303 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:31:17.413614 2573 flags.go:64] FLAG: --eviction-soft-grace-period="" Jul 09 15:31:17.419303 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:31:17.413617 2573 flags.go:64] FLAG: --exit-on-lock-contention="false" Jul 09 15:31:17.419914 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:31:17.413620 2573 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false" Jul 09 15:31:17.419914 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:31:17.413623 2573 flags.go:64] FLAG: --experimental-mounter-path="" Jul 09 15:31:17.419914 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:31:17.413626 2573 flags.go:64] FLAG: --fail-cgroupv1="false" Jul 09 15:31:17.419914 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:31:17.413629 2573 flags.go:64] FLAG: --fail-swap-on="true" Jul 09 15:31:17.419914 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:31:17.413632 2573 flags.go:64] FLAG: --feature-gates="" Jul 09 15:31:17.419914 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:31:17.413636 2573 flags.go:64] FLAG: --file-check-frequency="20s" Jul 09 15:31:17.419914 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:31:17.413639 2573 flags.go:64] FLAG: --global-housekeeping-interval="1m0s" Jul 09 15:31:17.419914 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:31:17.413642 2573 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge" Jul 09 15:31:17.419914 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:31:17.413645 2573 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1" Jul 09 15:31:17.419914 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:31:17.413648 2573 flags.go:64] FLAG: --healthz-port="10248" Jul 09 15:31:17.419914 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:31:17.413651 2573 flags.go:64] FLAG: --help="false" Jul 09 15:31:17.419914 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:31:17.413653 2573 flags.go:64] FLAG: --hostname-override="ip-10-0-133-183.ec2.internal" Jul 09 15:31:17.419914 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:31:17.413657 2573 flags.go:64] FLAG: --housekeeping-interval="10s" Jul 09 15:31:17.419914 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:31:17.413659 2573 flags.go:64] FLAG: --http-check-frequency="20s" Jul 09 15:31:17.419914 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:31:17.413662 2573 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins" Jul 09 15:31:17.419914 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:31:17.413666 2573 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml" Jul 09 15:31:17.419914 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:31:17.413669 2573 flags.go:64] FLAG: --image-gc-high-threshold="85" Jul 09 15:31:17.419914 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:31:17.413672 2573 flags.go:64] FLAG: --image-gc-low-threshold="80" Jul 09 15:31:17.419914 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:31:17.413676 2573 flags.go:64] FLAG: --image-service-endpoint="" Jul 09 15:31:17.419914 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:31:17.413679 2573 flags.go:64] FLAG: --kernel-memcg-notification="false" Jul 09 15:31:17.419914 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:31:17.413683 2573 flags.go:64] FLAG: --kube-api-burst="100" Jul 09 15:31:17.419914 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:31:17.413686 2573 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf" Jul 09 15:31:17.419914 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:31:17.413690 2573 flags.go:64] FLAG: --kube-api-qps="50" Jul 09 15:31:17.419914 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:31:17.413693 2573 flags.go:64] FLAG: --kube-reserved="" Jul 09 15:31:17.420584 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:31:17.413696 2573 flags.go:64] FLAG: --kube-reserved-cgroup="" Jul 09 15:31:17.420584 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:31:17.413698 2573 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig" Jul 09 15:31:17.420584 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:31:17.413701 2573 flags.go:64] FLAG: --kubelet-cgroups="" Jul 09 15:31:17.420584 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:31:17.413704 2573 flags.go:64] FLAG: --local-storage-capacity-isolation="true" Jul 09 15:31:17.420584 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:31:17.413707 2573 flags.go:64] FLAG: --lock-file="" Jul 09 15:31:17.420584 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:31:17.413710 2573 flags.go:64] FLAG: --log-cadvisor-usage="false" Jul 09 15:31:17.420584 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:31:17.413713 2573 flags.go:64] FLAG: --log-flush-frequency="5s" Jul 09 15:31:17.420584 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:31:17.413715 2573 flags.go:64] FLAG: --log-json-info-buffer-size="0" Jul 09 15:31:17.420584 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:31:17.413721 2573 flags.go:64] FLAG: --log-json-split-stream="false" Jul 09 15:31:17.420584 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:31:17.413724 2573 flags.go:64] FLAG: --log-text-info-buffer-size="0" Jul 09 15:31:17.420584 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:31:17.413726 2573 flags.go:64] FLAG: --log-text-split-stream="false" Jul 09 15:31:17.420584 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:31:17.413729 2573 flags.go:64] FLAG: --logging-format="text" Jul 09 15:31:17.420584 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:31:17.413732 2573 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id" Jul 09 15:31:17.420584 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:31:17.413735 2573 flags.go:64] FLAG: --make-iptables-util-chains="true" Jul 09 15:31:17.420584 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:31:17.413738 2573 flags.go:64] FLAG: --manifest-url="" Jul 09 15:31:17.420584 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:31:17.413741 2573 flags.go:64] FLAG: --manifest-url-header="" Jul 09 15:31:17.420584 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:31:17.413745 2573 flags.go:64] FLAG: --max-housekeeping-interval="15s" Jul 09 15:31:17.420584 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:31:17.413749 2573 flags.go:64] FLAG: --max-open-files="1000000" Jul 09 15:31:17.420584 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:31:17.413753 2573 flags.go:64] FLAG: --max-pods="110" Jul 09 15:31:17.420584 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:31:17.413756 2573 flags.go:64] FLAG: --maximum-dead-containers="-1" Jul 09 15:31:17.420584 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:31:17.413759 2573 flags.go:64] FLAG: --maximum-dead-containers-per-container="1" Jul 09 15:31:17.420584 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:31:17.413762 2573 flags.go:64] FLAG: --memory-manager-policy="None" Jul 09 15:31:17.420584 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:31:17.413765 2573 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s" Jul 09 15:31:17.420584 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:31:17.413768 2573 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s" Jul 09 15:31:17.420584 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:31:17.413771 2573 flags.go:64] FLAG: --node-ip="0.0.0.0" Jul 09 15:31:17.421216 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:31:17.413774 2573 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel" Jul 09 15:31:17.421216 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:31:17.413783 2573 flags.go:64] FLAG: --node-status-max-images="50" Jul 09 15:31:17.421216 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:31:17.413786 2573 flags.go:64] FLAG: --node-status-update-frequency="10s" Jul 09 15:31:17.421216 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:31:17.413789 2573 flags.go:64] FLAG: --oom-score-adj="-999" Jul 09 15:31:17.421216 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:31:17.413792 2573 flags.go:64] FLAG: --pod-cidr="" Jul 09 15:31:17.421216 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:31:17.413796 2573 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:b04c0b1f73af4d021dd8f0b51bfeb302c24482c093de5f2c43654ab56c506926" Jul 09 15:31:17.421216 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:31:17.413801 2573 flags.go:64] FLAG: --pod-manifest-path="" Jul 09 15:31:17.421216 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:31:17.413805 2573 flags.go:64] FLAG: --pod-max-pids="-1" Jul 09 15:31:17.421216 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:31:17.413808 2573 flags.go:64] FLAG: --pods-per-core="0" Jul 09 15:31:17.421216 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:31:17.413811 2573 flags.go:64] FLAG: --port="10250" Jul 09 15:31:17.421216 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:31:17.413814 2573 flags.go:64] FLAG: --protect-kernel-defaults="false" Jul 09 15:31:17.421216 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:31:17.413817 2573 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-049eafc2b77c0715f" Jul 09 15:31:17.421216 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:31:17.413820 2573 flags.go:64] FLAG: --qos-reserved="" Jul 09 15:31:17.421216 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:31:17.413823 2573 flags.go:64] FLAG: --read-only-port="10255" Jul 09 15:31:17.421216 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:31:17.413826 2573 flags.go:64] FLAG: --register-node="true" Jul 09 15:31:17.421216 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:31:17.413829 2573 flags.go:64] FLAG: --register-with-taints="" Jul 09 15:31:17.421216 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:31:17.413832 2573 flags.go:64] FLAG: --registry-burst="10" Jul 09 15:31:17.421216 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:31:17.413835 2573 flags.go:64] FLAG: --registry-qps="5" Jul 09 15:31:17.421216 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:31:17.413838 2573 flags.go:64] FLAG: --reserved-cpus="" Jul 09 15:31:17.421216 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:31:17.413840 2573 flags.go:64] FLAG: --reserved-memory="" Jul 09 15:31:17.421216 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:31:17.413844 2573 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf" Jul 09 15:31:17.421216 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:31:17.413848 2573 flags.go:64] FLAG: --root-dir="/var/lib/kubelet" Jul 09 15:31:17.421216 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:31:17.413851 2573 flags.go:64] FLAG: --rotate-certificates="false" Jul 09 15:31:17.421216 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:31:17.413854 2573 flags.go:64] FLAG: --rotate-server-certificates="false" Jul 09 15:31:17.421216 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:31:17.413856 2573 flags.go:64] FLAG: --runonce="false" Jul 09 15:31:17.421823 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:31:17.413859 2573 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service" Jul 09 15:31:17.421823 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:31:17.413862 2573 flags.go:64] FLAG: --runtime-request-timeout="2m0s" Jul 09 15:31:17.421823 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:31:17.413865 2573 flags.go:64] FLAG: --seccomp-default="false" Jul 09 15:31:17.421823 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:31:17.413868 2573 flags.go:64] FLAG: --serialize-image-pulls="true" Jul 09 15:31:17.421823 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:31:17.413871 2573 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s" Jul 09 15:31:17.421823 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:31:17.413874 2573 flags.go:64] FLAG: --storage-driver-db="cadvisor" Jul 09 15:31:17.421823 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:31:17.413877 2573 flags.go:64] FLAG: --storage-driver-host="localhost:8086" Jul 09 15:31:17.421823 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:31:17.413880 2573 flags.go:64] FLAG: --storage-driver-password="root" Jul 09 15:31:17.421823 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:31:17.413885 2573 flags.go:64] FLAG: --storage-driver-secure="false" Jul 09 15:31:17.421823 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:31:17.413888 2573 flags.go:64] FLAG: --storage-driver-table="stats" Jul 09 15:31:17.421823 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:31:17.413890 2573 flags.go:64] FLAG: --storage-driver-user="root" Jul 09 15:31:17.421823 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:31:17.413893 2573 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s" Jul 09 15:31:17.421823 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:31:17.413896 2573 flags.go:64] FLAG: --sync-frequency="1m0s" Jul 09 15:31:17.421823 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:31:17.413900 2573 flags.go:64] FLAG: --system-cgroups="" Jul 09 15:31:17.421823 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:31:17.413903 2573 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi" Jul 09 15:31:17.421823 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:31:17.413909 2573 flags.go:64] FLAG: --system-reserved-cgroup="" Jul 09 15:31:17.421823 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:31:17.413912 2573 flags.go:64] FLAG: --tls-cert-file="" Jul 09 15:31:17.421823 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:31:17.413915 2573 flags.go:64] FLAG: --tls-cipher-suites="[]" Jul 09 15:31:17.421823 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:31:17.413919 2573 flags.go:64] FLAG: --tls-min-version="" Jul 09 15:31:17.421823 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:31:17.413922 2573 flags.go:64] FLAG: --tls-private-key-file="" Jul 09 15:31:17.421823 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:31:17.413925 2573 flags.go:64] FLAG: --topology-manager-policy="none" Jul 09 15:31:17.421823 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:31:17.413928 2573 flags.go:64] FLAG: --topology-manager-policy-options="" Jul 09 15:31:17.421823 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:31:17.413931 2573 flags.go:64] FLAG: --topology-manager-scope="container" Jul 09 15:31:17.421823 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:31:17.413934 2573 flags.go:64] FLAG: --v="2" Jul 09 15:31:17.421823 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:31:17.413938 2573 flags.go:64] FLAG: --version="false" Jul 09 15:31:17.422476 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:31:17.413942 2573 flags.go:64] FLAG: --vmodule="" Jul 09 15:31:17.422476 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:31:17.413946 2573 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec" Jul 09 15:31:17.422476 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:31:17.413949 2573 flags.go:64] FLAG: --volume-stats-agg-period="1m0s" Jul 09 15:31:17.422476 ip-10-0-133-183 kubenswrapper[2573]: W0709 15:31:17.414083 2573 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 09 15:31:17.422476 ip-10-0-133-183 kubenswrapper[2573]: W0709 15:31:17.414088 2573 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 09 15:31:17.422476 ip-10-0-133-183 kubenswrapper[2573]: W0709 15:31:17.414092 2573 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 09 15:31:17.422476 ip-10-0-133-183 kubenswrapper[2573]: W0709 15:31:17.414095 2573 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 09 15:31:17.422476 ip-10-0-133-183 kubenswrapper[2573]: W0709 15:31:17.414098 2573 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 09 15:31:17.422476 ip-10-0-133-183 kubenswrapper[2573]: W0709 15:31:17.414101 2573 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 09 15:31:17.422476 ip-10-0-133-183 kubenswrapper[2573]: W0709 15:31:17.414104 2573 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 09 15:31:17.422476 ip-10-0-133-183 kubenswrapper[2573]: W0709 15:31:17.414107 2573 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 09 15:31:17.422476 ip-10-0-133-183 kubenswrapper[2573]: W0709 15:31:17.414110 2573 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 09 15:31:17.422476 ip-10-0-133-183 kubenswrapper[2573]: W0709 15:31:17.414113 2573 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 09 15:31:17.422476 ip-10-0-133-183 kubenswrapper[2573]: W0709 15:31:17.414117 2573 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 09 15:31:17.422476 ip-10-0-133-183 kubenswrapper[2573]: W0709 15:31:17.414120 2573 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 09 15:31:17.422476 ip-10-0-133-183 kubenswrapper[2573]: W0709 15:31:17.414124 2573 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 09 15:31:17.422476 ip-10-0-133-183 kubenswrapper[2573]: W0709 15:31:17.414129 2573 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 09 15:31:17.422476 ip-10-0-133-183 kubenswrapper[2573]: W0709 15:31:17.414131 2573 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 09 15:31:17.422476 ip-10-0-133-183 kubenswrapper[2573]: W0709 15:31:17.414134 2573 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 09 15:31:17.422944 ip-10-0-133-183 kubenswrapper[2573]: W0709 15:31:17.414137 2573 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 09 15:31:17.422944 ip-10-0-133-183 kubenswrapper[2573]: W0709 15:31:17.414139 2573 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 09 15:31:17.422944 ip-10-0-133-183 kubenswrapper[2573]: W0709 15:31:17.414143 2573 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 09 15:31:17.422944 ip-10-0-133-183 kubenswrapper[2573]: W0709 15:31:17.414146 2573 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 09 15:31:17.422944 ip-10-0-133-183 kubenswrapper[2573]: W0709 15:31:17.414148 2573 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 09 15:31:17.422944 ip-10-0-133-183 kubenswrapper[2573]: W0709 15:31:17.414151 2573 feature_gate.go:328] unrecognized feature gate: Example2 Jul 09 15:31:17.422944 ip-10-0-133-183 kubenswrapper[2573]: W0709 15:31:17.414153 2573 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 09 15:31:17.422944 ip-10-0-133-183 kubenswrapper[2573]: W0709 15:31:17.414156 2573 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 09 15:31:17.422944 ip-10-0-133-183 kubenswrapper[2573]: W0709 15:31:17.414159 2573 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 09 15:31:17.422944 ip-10-0-133-183 kubenswrapper[2573]: W0709 15:31:17.414161 2573 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 09 15:31:17.422944 ip-10-0-133-183 kubenswrapper[2573]: W0709 15:31:17.414164 2573 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 09 15:31:17.422944 ip-10-0-133-183 kubenswrapper[2573]: W0709 15:31:17.414166 2573 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 09 15:31:17.422944 ip-10-0-133-183 kubenswrapper[2573]: W0709 15:31:17.414169 2573 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 09 15:31:17.422944 ip-10-0-133-183 kubenswrapper[2573]: W0709 15:31:17.414172 2573 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 09 15:31:17.422944 ip-10-0-133-183 kubenswrapper[2573]: W0709 15:31:17.414175 2573 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 09 15:31:17.422944 ip-10-0-133-183 kubenswrapper[2573]: W0709 15:31:17.414177 2573 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 09 15:31:17.422944 ip-10-0-133-183 kubenswrapper[2573]: W0709 15:31:17.414180 2573 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 09 15:31:17.422944 ip-10-0-133-183 kubenswrapper[2573]: W0709 15:31:17.414182 2573 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 09 15:31:17.422944 ip-10-0-133-183 kubenswrapper[2573]: W0709 15:31:17.414185 2573 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 09 15:31:17.422944 ip-10-0-133-183 kubenswrapper[2573]: W0709 15:31:17.414187 2573 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 09 15:31:17.423450 ip-10-0-133-183 kubenswrapper[2573]: W0709 15:31:17.414190 2573 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 09 15:31:17.423450 ip-10-0-133-183 kubenswrapper[2573]: W0709 15:31:17.414192 2573 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 09 15:31:17.423450 ip-10-0-133-183 kubenswrapper[2573]: W0709 15:31:17.414195 2573 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 09 15:31:17.423450 ip-10-0-133-183 kubenswrapper[2573]: W0709 15:31:17.414197 2573 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 09 15:31:17.423450 ip-10-0-133-183 kubenswrapper[2573]: W0709 15:31:17.414200 2573 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 09 15:31:17.423450 ip-10-0-133-183 kubenswrapper[2573]: W0709 15:31:17.414202 2573 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 09 15:31:17.423450 ip-10-0-133-183 kubenswrapper[2573]: W0709 15:31:17.414204 2573 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 09 15:31:17.423450 ip-10-0-133-183 kubenswrapper[2573]: W0709 15:31:17.414207 2573 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 09 15:31:17.423450 ip-10-0-133-183 kubenswrapper[2573]: W0709 15:31:17.414211 2573 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 09 15:31:17.423450 ip-10-0-133-183 kubenswrapper[2573]: W0709 15:31:17.414215 2573 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 09 15:31:17.423450 ip-10-0-133-183 kubenswrapper[2573]: W0709 15:31:17.414218 2573 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 09 15:31:17.423450 ip-10-0-133-183 kubenswrapper[2573]: W0709 15:31:17.414220 2573 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 09 15:31:17.423450 ip-10-0-133-183 kubenswrapper[2573]: W0709 15:31:17.414223 2573 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 09 15:31:17.423450 ip-10-0-133-183 kubenswrapper[2573]: W0709 15:31:17.414225 2573 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 09 15:31:17.423450 ip-10-0-133-183 kubenswrapper[2573]: W0709 15:31:17.414228 2573 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 09 15:31:17.423450 ip-10-0-133-183 kubenswrapper[2573]: W0709 15:31:17.414232 2573 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 09 15:31:17.423450 ip-10-0-133-183 kubenswrapper[2573]: W0709 15:31:17.414234 2573 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 09 15:31:17.423450 ip-10-0-133-183 kubenswrapper[2573]: W0709 15:31:17.414237 2573 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 09 15:31:17.423450 ip-10-0-133-183 kubenswrapper[2573]: W0709 15:31:17.414241 2573 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 09 15:31:17.423914 ip-10-0-133-183 kubenswrapper[2573]: W0709 15:31:17.414245 2573 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 09 15:31:17.423914 ip-10-0-133-183 kubenswrapper[2573]: W0709 15:31:17.414247 2573 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 09 15:31:17.423914 ip-10-0-133-183 kubenswrapper[2573]: W0709 15:31:17.414250 2573 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 09 15:31:17.423914 ip-10-0-133-183 kubenswrapper[2573]: W0709 15:31:17.414253 2573 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 09 15:31:17.423914 ip-10-0-133-183 kubenswrapper[2573]: W0709 15:31:17.414256 2573 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 09 15:31:17.423914 ip-10-0-133-183 kubenswrapper[2573]: W0709 15:31:17.414259 2573 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 09 15:31:17.423914 ip-10-0-133-183 kubenswrapper[2573]: W0709 15:31:17.414261 2573 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 09 15:31:17.423914 ip-10-0-133-183 kubenswrapper[2573]: W0709 15:31:17.414263 2573 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 09 15:31:17.423914 ip-10-0-133-183 kubenswrapper[2573]: W0709 15:31:17.414266 2573 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 09 15:31:17.423914 ip-10-0-133-183 kubenswrapper[2573]: W0709 15:31:17.414269 2573 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 09 15:31:17.423914 ip-10-0-133-183 kubenswrapper[2573]: W0709 15:31:17.414271 2573 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 09 15:31:17.423914 ip-10-0-133-183 kubenswrapper[2573]: W0709 15:31:17.414275 2573 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 09 15:31:17.423914 ip-10-0-133-183 kubenswrapper[2573]: W0709 15:31:17.414278 2573 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 09 15:31:17.423914 ip-10-0-133-183 kubenswrapper[2573]: W0709 15:31:17.414281 2573 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 09 15:31:17.423914 ip-10-0-133-183 kubenswrapper[2573]: W0709 15:31:17.414284 2573 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 09 15:31:17.423914 ip-10-0-133-183 kubenswrapper[2573]: W0709 15:31:17.414286 2573 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 09 15:31:17.423914 ip-10-0-133-183 kubenswrapper[2573]: W0709 15:31:17.414289 2573 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 09 15:31:17.423914 ip-10-0-133-183 kubenswrapper[2573]: W0709 15:31:17.414291 2573 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 09 15:31:17.423914 ip-10-0-133-183 kubenswrapper[2573]: W0709 15:31:17.414294 2573 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 09 15:31:17.423914 ip-10-0-133-183 kubenswrapper[2573]: W0709 15:31:17.414296 2573 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 09 15:31:17.424506 ip-10-0-133-183 kubenswrapper[2573]: W0709 15:31:17.414299 2573 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 09 15:31:17.424506 ip-10-0-133-183 kubenswrapper[2573]: W0709 15:31:17.414302 2573 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 09 15:31:17.424506 ip-10-0-133-183 kubenswrapper[2573]: W0709 15:31:17.414307 2573 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 09 15:31:17.424506 ip-10-0-133-183 kubenswrapper[2573]: W0709 15:31:17.414309 2573 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 09 15:31:17.424506 ip-10-0-133-183 kubenswrapper[2573]: W0709 15:31:17.414312 2573 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 09 15:31:17.424506 ip-10-0-133-183 kubenswrapper[2573]: W0709 15:31:17.414315 2573 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 09 15:31:17.424506 ip-10-0-133-183 kubenswrapper[2573]: W0709 15:31:17.414318 2573 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 09 15:31:17.424506 ip-10-0-133-183 kubenswrapper[2573]: W0709 15:31:17.414321 2573 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 09 15:31:17.424506 ip-10-0-133-183 kubenswrapper[2573]: W0709 15:31:17.414323 2573 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 09 15:31:17.424506 ip-10-0-133-183 kubenswrapper[2573]: W0709 15:31:17.414326 2573 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 09 15:31:17.424506 ip-10-0-133-183 kubenswrapper[2573]: W0709 15:31:17.414329 2573 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 09 15:31:17.424506 ip-10-0-133-183 kubenswrapper[2573]: W0709 15:31:17.414332 2573 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 09 15:31:17.424506 ip-10-0-133-183 kubenswrapper[2573]: W0709 15:31:17.414334 2573 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 09 15:31:17.424506 ip-10-0-133-183 kubenswrapper[2573]: W0709 15:31:17.414337 2573 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 09 15:31:17.424506 ip-10-0-133-183 kubenswrapper[2573]: W0709 15:31:17.414339 2573 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 09 15:31:17.424506 ip-10-0-133-183 kubenswrapper[2573]: W0709 15:31:17.414342 2573 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 09 15:31:17.424506 ip-10-0-133-183 kubenswrapper[2573]: W0709 15:31:17.414345 2573 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 09 15:31:17.424506 ip-10-0-133-183 kubenswrapper[2573]: W0709 15:31:17.414347 2573 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 09 15:31:17.424506 ip-10-0-133-183 kubenswrapper[2573]: W0709 15:31:17.414350 2573 feature_gate.go:328] unrecognized feature gate: Example Jul 09 15:31:17.424506 ip-10-0-133-183 kubenswrapper[2573]: W0709 15:31:17.414352 2573 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 09 15:31:17.425022 ip-10-0-133-183 kubenswrapper[2573]: W0709 15:31:17.414355 2573 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 09 15:31:17.425022 ip-10-0-133-183 kubenswrapper[2573]: W0709 15:31:17.414357 2573 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 09 15:31:17.425022 ip-10-0-133-183 kubenswrapper[2573]: W0709 15:31:17.414359 2573 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 09 15:31:17.425022 ip-10-0-133-183 kubenswrapper[2573]: W0709 15:31:17.414362 2573 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 09 15:31:17.425022 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:31:17.415046 2573 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jul 09 15:31:17.425022 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:31:17.422276 2573 server.go:529] "Kubelet version" kubeletVersion="v1.34.9" Jul 09 15:31:17.425022 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:31:17.422295 2573 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK="" Jul 09 15:31:17.425022 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:31:17.423057 2573 watchdog_linux.go:95] "Systemd watchdog is not enabled" Jul 09 15:31:17.425022 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:31:17.423066 2573 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started." Jul 09 15:31:17.425022 ip-10-0-133-183 kubenswrapper[2573]: W0709 15:31:17.423127 2573 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 09 15:31:17.425022 ip-10-0-133-183 kubenswrapper[2573]: W0709 15:31:17.423133 2573 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 09 15:31:17.425022 ip-10-0-133-183 kubenswrapper[2573]: W0709 15:31:17.423137 2573 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 09 15:31:17.425022 ip-10-0-133-183 kubenswrapper[2573]: W0709 15:31:17.423140 2573 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 09 15:31:17.425022 ip-10-0-133-183 kubenswrapper[2573]: W0709 15:31:17.423144 2573 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 09 15:31:17.425022 ip-10-0-133-183 kubenswrapper[2573]: W0709 15:31:17.423147 2573 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 09 15:31:17.425399 ip-10-0-133-183 kubenswrapper[2573]: W0709 15:31:17.423150 2573 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 09 15:31:17.425399 ip-10-0-133-183 kubenswrapper[2573]: W0709 15:31:17.423153 2573 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 09 15:31:17.425399 ip-10-0-133-183 kubenswrapper[2573]: W0709 15:31:17.423156 2573 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 09 15:31:17.425399 ip-10-0-133-183 kubenswrapper[2573]: W0709 15:31:17.423159 2573 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 09 15:31:17.425399 ip-10-0-133-183 kubenswrapper[2573]: W0709 15:31:17.423161 2573 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 09 15:31:17.425399 ip-10-0-133-183 kubenswrapper[2573]: W0709 15:31:17.423164 2573 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 09 15:31:17.425399 ip-10-0-133-183 kubenswrapper[2573]: W0709 15:31:17.423167 2573 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 09 15:31:17.425399 ip-10-0-133-183 kubenswrapper[2573]: W0709 15:31:17.423169 2573 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 09 15:31:17.425399 ip-10-0-133-183 kubenswrapper[2573]: W0709 15:31:17.423172 2573 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 09 15:31:17.425399 ip-10-0-133-183 kubenswrapper[2573]: W0709 15:31:17.423175 2573 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 09 15:31:17.425399 ip-10-0-133-183 kubenswrapper[2573]: W0709 15:31:17.423178 2573 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 09 15:31:17.425399 ip-10-0-133-183 kubenswrapper[2573]: W0709 15:31:17.423180 2573 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 09 15:31:17.425399 ip-10-0-133-183 kubenswrapper[2573]: W0709 15:31:17.423184 2573 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 09 15:31:17.425399 ip-10-0-133-183 kubenswrapper[2573]: W0709 15:31:17.423188 2573 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 09 15:31:17.425399 ip-10-0-133-183 kubenswrapper[2573]: W0709 15:31:17.423191 2573 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 09 15:31:17.425399 ip-10-0-133-183 kubenswrapper[2573]: W0709 15:31:17.423194 2573 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 09 15:31:17.425399 ip-10-0-133-183 kubenswrapper[2573]: W0709 15:31:17.423197 2573 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 09 15:31:17.425399 ip-10-0-133-183 kubenswrapper[2573]: W0709 15:31:17.423200 2573 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 09 15:31:17.425399 ip-10-0-133-183 kubenswrapper[2573]: W0709 15:31:17.423202 2573 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 09 15:31:17.425399 ip-10-0-133-183 kubenswrapper[2573]: W0709 15:31:17.423205 2573 feature_gate.go:328] unrecognized feature gate: Example Jul 09 15:31:17.425888 ip-10-0-133-183 kubenswrapper[2573]: W0709 15:31:17.423208 2573 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 09 15:31:17.425888 ip-10-0-133-183 kubenswrapper[2573]: W0709 15:31:17.423211 2573 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 09 15:31:17.425888 ip-10-0-133-183 kubenswrapper[2573]: W0709 15:31:17.423213 2573 feature_gate.go:328] unrecognized feature gate: Example2 Jul 09 15:31:17.425888 ip-10-0-133-183 kubenswrapper[2573]: W0709 15:31:17.423217 2573 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 09 15:31:17.425888 ip-10-0-133-183 kubenswrapper[2573]: W0709 15:31:17.423220 2573 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 09 15:31:17.425888 ip-10-0-133-183 kubenswrapper[2573]: W0709 15:31:17.423223 2573 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 09 15:31:17.425888 ip-10-0-133-183 kubenswrapper[2573]: W0709 15:31:17.423226 2573 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 09 15:31:17.425888 ip-10-0-133-183 kubenswrapper[2573]: W0709 15:31:17.423229 2573 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 09 15:31:17.425888 ip-10-0-133-183 kubenswrapper[2573]: W0709 15:31:17.423232 2573 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 09 15:31:17.425888 ip-10-0-133-183 kubenswrapper[2573]: W0709 15:31:17.423234 2573 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 09 15:31:17.425888 ip-10-0-133-183 kubenswrapper[2573]: W0709 15:31:17.423237 2573 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 09 15:31:17.425888 ip-10-0-133-183 kubenswrapper[2573]: W0709 15:31:17.423239 2573 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 09 15:31:17.425888 ip-10-0-133-183 kubenswrapper[2573]: W0709 15:31:17.423242 2573 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 09 15:31:17.425888 ip-10-0-133-183 kubenswrapper[2573]: W0709 15:31:17.423244 2573 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 09 15:31:17.425888 ip-10-0-133-183 kubenswrapper[2573]: W0709 15:31:17.423247 2573 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 09 15:31:17.425888 ip-10-0-133-183 kubenswrapper[2573]: W0709 15:31:17.423249 2573 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 09 15:31:17.425888 ip-10-0-133-183 kubenswrapper[2573]: W0709 15:31:17.423252 2573 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 09 15:31:17.425888 ip-10-0-133-183 kubenswrapper[2573]: W0709 15:31:17.423254 2573 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 09 15:31:17.425888 ip-10-0-133-183 kubenswrapper[2573]: W0709 15:31:17.423257 2573 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 09 15:31:17.425888 ip-10-0-133-183 kubenswrapper[2573]: W0709 15:31:17.423259 2573 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 09 15:31:17.426372 ip-10-0-133-183 kubenswrapper[2573]: W0709 15:31:17.423262 2573 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 09 15:31:17.426372 ip-10-0-133-183 kubenswrapper[2573]: W0709 15:31:17.423264 2573 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 09 15:31:17.426372 ip-10-0-133-183 kubenswrapper[2573]: W0709 15:31:17.423267 2573 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 09 15:31:17.426372 ip-10-0-133-183 kubenswrapper[2573]: W0709 15:31:17.423269 2573 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 09 15:31:17.426372 ip-10-0-133-183 kubenswrapper[2573]: W0709 15:31:17.423271 2573 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 09 15:31:17.426372 ip-10-0-133-183 kubenswrapper[2573]: W0709 15:31:17.423274 2573 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 09 15:31:17.426372 ip-10-0-133-183 kubenswrapper[2573]: W0709 15:31:17.423277 2573 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 09 15:31:17.426372 ip-10-0-133-183 kubenswrapper[2573]: W0709 15:31:17.423279 2573 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 09 15:31:17.426372 ip-10-0-133-183 kubenswrapper[2573]: W0709 15:31:17.423282 2573 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 09 15:31:17.426372 ip-10-0-133-183 kubenswrapper[2573]: W0709 15:31:17.423284 2573 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 09 15:31:17.426372 ip-10-0-133-183 kubenswrapper[2573]: W0709 15:31:17.423287 2573 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 09 15:31:17.426372 ip-10-0-133-183 kubenswrapper[2573]: W0709 15:31:17.423289 2573 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 09 15:31:17.426372 ip-10-0-133-183 kubenswrapper[2573]: W0709 15:31:17.423292 2573 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 09 15:31:17.426372 ip-10-0-133-183 kubenswrapper[2573]: W0709 15:31:17.423295 2573 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 09 15:31:17.426372 ip-10-0-133-183 kubenswrapper[2573]: W0709 15:31:17.423297 2573 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 09 15:31:17.426372 ip-10-0-133-183 kubenswrapper[2573]: W0709 15:31:17.423300 2573 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 09 15:31:17.426372 ip-10-0-133-183 kubenswrapper[2573]: W0709 15:31:17.423305 2573 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 09 15:31:17.426372 ip-10-0-133-183 kubenswrapper[2573]: W0709 15:31:17.423307 2573 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 09 15:31:17.426372 ip-10-0-133-183 kubenswrapper[2573]: W0709 15:31:17.423310 2573 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 09 15:31:17.426372 ip-10-0-133-183 kubenswrapper[2573]: W0709 15:31:17.423312 2573 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 09 15:31:17.426869 ip-10-0-133-183 kubenswrapper[2573]: W0709 15:31:17.423315 2573 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 09 15:31:17.426869 ip-10-0-133-183 kubenswrapper[2573]: W0709 15:31:17.423318 2573 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 09 15:31:17.426869 ip-10-0-133-183 kubenswrapper[2573]: W0709 15:31:17.423320 2573 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 09 15:31:17.426869 ip-10-0-133-183 kubenswrapper[2573]: W0709 15:31:17.423323 2573 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 09 15:31:17.426869 ip-10-0-133-183 kubenswrapper[2573]: W0709 15:31:17.423325 2573 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 09 15:31:17.426869 ip-10-0-133-183 kubenswrapper[2573]: W0709 15:31:17.423328 2573 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 09 15:31:17.426869 ip-10-0-133-183 kubenswrapper[2573]: W0709 15:31:17.423330 2573 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 09 15:31:17.426869 ip-10-0-133-183 kubenswrapper[2573]: W0709 15:31:17.423333 2573 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 09 15:31:17.426869 ip-10-0-133-183 kubenswrapper[2573]: W0709 15:31:17.423335 2573 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 09 15:31:17.426869 ip-10-0-133-183 kubenswrapper[2573]: W0709 15:31:17.423338 2573 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 09 15:31:17.426869 ip-10-0-133-183 kubenswrapper[2573]: W0709 15:31:17.423340 2573 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 09 15:31:17.426869 ip-10-0-133-183 kubenswrapper[2573]: W0709 15:31:17.423343 2573 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 09 15:31:17.426869 ip-10-0-133-183 kubenswrapper[2573]: W0709 15:31:17.423345 2573 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 09 15:31:17.426869 ip-10-0-133-183 kubenswrapper[2573]: W0709 15:31:17.423347 2573 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 09 15:31:17.426869 ip-10-0-133-183 kubenswrapper[2573]: W0709 15:31:17.423351 2573 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 09 15:31:17.426869 ip-10-0-133-183 kubenswrapper[2573]: W0709 15:31:17.423355 2573 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 09 15:31:17.426869 ip-10-0-133-183 kubenswrapper[2573]: W0709 15:31:17.423358 2573 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 09 15:31:17.426869 ip-10-0-133-183 kubenswrapper[2573]: W0709 15:31:17.423360 2573 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 09 15:31:17.426869 ip-10-0-133-183 kubenswrapper[2573]: W0709 15:31:17.423363 2573 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 09 15:31:17.426869 ip-10-0-133-183 kubenswrapper[2573]: W0709 15:31:17.423366 2573 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 09 15:31:17.427370 ip-10-0-133-183 kubenswrapper[2573]: W0709 15:31:17.423369 2573 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 09 15:31:17.427370 ip-10-0-133-183 kubenswrapper[2573]: W0709 15:31:17.423371 2573 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 09 15:31:17.427370 ip-10-0-133-183 kubenswrapper[2573]: W0709 15:31:17.423373 2573 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 09 15:31:17.427370 ip-10-0-133-183 kubenswrapper[2573]: W0709 15:31:17.423377 2573 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 09 15:31:17.427370 ip-10-0-133-183 kubenswrapper[2573]: W0709 15:31:17.423380 2573 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 09 15:31:17.427370 ip-10-0-133-183 kubenswrapper[2573]: W0709 15:31:17.423383 2573 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 09 15:31:17.427370 ip-10-0-133-183 kubenswrapper[2573]: W0709 15:31:17.423386 2573 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 09 15:31:17.427370 ip-10-0-133-183 kubenswrapper[2573]: W0709 15:31:17.423389 2573 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 09 15:31:17.427370 ip-10-0-133-183 kubenswrapper[2573]: W0709 15:31:17.423392 2573 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 09 15:31:17.427370 ip-10-0-133-183 kubenswrapper[2573]: W0709 15:31:17.423395 2573 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 09 15:31:17.427370 ip-10-0-133-183 kubenswrapper[2573]: W0709 15:31:17.423398 2573 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 09 15:31:17.427370 ip-10-0-133-183 kubenswrapper[2573]: W0709 15:31:17.423401 2573 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 09 15:31:17.427370 ip-10-0-133-183 kubenswrapper[2573]: W0709 15:31:17.423403 2573 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 09 15:31:17.427370 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:31:17.423409 2573 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jul 09 15:31:17.427720 ip-10-0-133-183 kubenswrapper[2573]: W0709 15:31:17.423528 2573 feature_gate.go:328] unrecognized feature gate: Example Jul 09 15:31:17.427720 ip-10-0-133-183 kubenswrapper[2573]: W0709 15:31:17.423534 2573 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 09 15:31:17.427720 ip-10-0-133-183 kubenswrapper[2573]: W0709 15:31:17.423536 2573 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 09 15:31:17.427720 ip-10-0-133-183 kubenswrapper[2573]: W0709 15:31:17.423539 2573 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 09 15:31:17.427720 ip-10-0-133-183 kubenswrapper[2573]: W0709 15:31:17.423542 2573 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 09 15:31:17.427720 ip-10-0-133-183 kubenswrapper[2573]: W0709 15:31:17.423545 2573 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 09 15:31:17.427720 ip-10-0-133-183 kubenswrapper[2573]: W0709 15:31:17.423547 2573 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 09 15:31:17.427720 ip-10-0-133-183 kubenswrapper[2573]: W0709 15:31:17.423550 2573 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 09 15:31:17.427720 ip-10-0-133-183 kubenswrapper[2573]: W0709 15:31:17.423553 2573 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 09 15:31:17.427720 ip-10-0-133-183 kubenswrapper[2573]: W0709 15:31:17.423556 2573 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 09 15:31:17.427720 ip-10-0-133-183 kubenswrapper[2573]: W0709 15:31:17.423560 2573 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 09 15:31:17.427720 ip-10-0-133-183 kubenswrapper[2573]: W0709 15:31:17.423563 2573 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 09 15:31:17.427720 ip-10-0-133-183 kubenswrapper[2573]: W0709 15:31:17.423566 2573 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 09 15:31:17.427720 ip-10-0-133-183 kubenswrapper[2573]: W0709 15:31:17.423568 2573 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 09 15:31:17.427720 ip-10-0-133-183 kubenswrapper[2573]: W0709 15:31:17.423571 2573 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 09 15:31:17.427720 ip-10-0-133-183 kubenswrapper[2573]: W0709 15:31:17.423574 2573 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 09 15:31:17.427720 ip-10-0-133-183 kubenswrapper[2573]: W0709 15:31:17.423576 2573 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 09 15:31:17.427720 ip-10-0-133-183 kubenswrapper[2573]: W0709 15:31:17.423579 2573 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 09 15:31:17.427720 ip-10-0-133-183 kubenswrapper[2573]: W0709 15:31:17.423581 2573 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 09 15:31:17.428245 ip-10-0-133-183 kubenswrapper[2573]: W0709 15:31:17.423584 2573 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 09 15:31:17.428245 ip-10-0-133-183 kubenswrapper[2573]: W0709 15:31:17.423586 2573 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 09 15:31:17.428245 ip-10-0-133-183 kubenswrapper[2573]: W0709 15:31:17.423589 2573 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 09 15:31:17.428245 ip-10-0-133-183 kubenswrapper[2573]: W0709 15:31:17.423591 2573 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 09 15:31:17.428245 ip-10-0-133-183 kubenswrapper[2573]: W0709 15:31:17.423594 2573 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 09 15:31:17.428245 ip-10-0-133-183 kubenswrapper[2573]: W0709 15:31:17.423597 2573 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 09 15:31:17.428245 ip-10-0-133-183 kubenswrapper[2573]: W0709 15:31:17.423600 2573 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 09 15:31:17.428245 ip-10-0-133-183 kubenswrapper[2573]: W0709 15:31:17.423602 2573 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 09 15:31:17.428245 ip-10-0-133-183 kubenswrapper[2573]: W0709 15:31:17.423605 2573 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 09 15:31:17.428245 ip-10-0-133-183 kubenswrapper[2573]: W0709 15:31:17.423608 2573 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 09 15:31:17.428245 ip-10-0-133-183 kubenswrapper[2573]: W0709 15:31:17.423611 2573 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 09 15:31:17.428245 ip-10-0-133-183 kubenswrapper[2573]: W0709 15:31:17.423614 2573 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 09 15:31:17.428245 ip-10-0-133-183 kubenswrapper[2573]: W0709 15:31:17.423618 2573 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 09 15:31:17.428245 ip-10-0-133-183 kubenswrapper[2573]: W0709 15:31:17.423621 2573 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 09 15:31:17.428245 ip-10-0-133-183 kubenswrapper[2573]: W0709 15:31:17.423624 2573 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 09 15:31:17.428245 ip-10-0-133-183 kubenswrapper[2573]: W0709 15:31:17.423627 2573 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 09 15:31:17.428245 ip-10-0-133-183 kubenswrapper[2573]: W0709 15:31:17.423630 2573 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 09 15:31:17.428245 ip-10-0-133-183 kubenswrapper[2573]: W0709 15:31:17.423633 2573 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 09 15:31:17.428245 ip-10-0-133-183 kubenswrapper[2573]: W0709 15:31:17.423635 2573 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 09 15:31:17.428713 ip-10-0-133-183 kubenswrapper[2573]: W0709 15:31:17.423638 2573 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 09 15:31:17.428713 ip-10-0-133-183 kubenswrapper[2573]: W0709 15:31:17.423640 2573 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 09 15:31:17.428713 ip-10-0-133-183 kubenswrapper[2573]: W0709 15:31:17.423643 2573 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 09 15:31:17.428713 ip-10-0-133-183 kubenswrapper[2573]: W0709 15:31:17.423645 2573 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 09 15:31:17.428713 ip-10-0-133-183 kubenswrapper[2573]: W0709 15:31:17.423648 2573 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 09 15:31:17.428713 ip-10-0-133-183 kubenswrapper[2573]: W0709 15:31:17.423650 2573 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 09 15:31:17.428713 ip-10-0-133-183 kubenswrapper[2573]: W0709 15:31:17.423652 2573 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 09 15:31:17.428713 ip-10-0-133-183 kubenswrapper[2573]: W0709 15:31:17.423655 2573 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 09 15:31:17.428713 ip-10-0-133-183 kubenswrapper[2573]: W0709 15:31:17.423657 2573 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 09 15:31:17.428713 ip-10-0-133-183 kubenswrapper[2573]: W0709 15:31:17.423660 2573 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 09 15:31:17.428713 ip-10-0-133-183 kubenswrapper[2573]: W0709 15:31:17.423662 2573 feature_gate.go:328] unrecognized feature gate: Example2 Jul 09 15:31:17.428713 ip-10-0-133-183 kubenswrapper[2573]: W0709 15:31:17.423665 2573 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 09 15:31:17.428713 ip-10-0-133-183 kubenswrapper[2573]: W0709 15:31:17.423667 2573 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 09 15:31:17.428713 ip-10-0-133-183 kubenswrapper[2573]: W0709 15:31:17.423669 2573 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 09 15:31:17.428713 ip-10-0-133-183 kubenswrapper[2573]: W0709 15:31:17.423672 2573 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 09 15:31:17.428713 ip-10-0-133-183 kubenswrapper[2573]: W0709 15:31:17.423674 2573 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 09 15:31:17.428713 ip-10-0-133-183 kubenswrapper[2573]: W0709 15:31:17.423677 2573 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 09 15:31:17.428713 ip-10-0-133-183 kubenswrapper[2573]: W0709 15:31:17.423679 2573 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 09 15:31:17.428713 ip-10-0-133-183 kubenswrapper[2573]: W0709 15:31:17.423683 2573 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 09 15:31:17.428713 ip-10-0-133-183 kubenswrapper[2573]: W0709 15:31:17.423685 2573 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 09 15:31:17.429209 ip-10-0-133-183 kubenswrapper[2573]: W0709 15:31:17.423688 2573 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 09 15:31:17.429209 ip-10-0-133-183 kubenswrapper[2573]: W0709 15:31:17.423690 2573 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 09 15:31:17.429209 ip-10-0-133-183 kubenswrapper[2573]: W0709 15:31:17.423692 2573 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 09 15:31:17.429209 ip-10-0-133-183 kubenswrapper[2573]: W0709 15:31:17.423696 2573 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 09 15:31:17.429209 ip-10-0-133-183 kubenswrapper[2573]: W0709 15:31:17.423698 2573 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 09 15:31:17.429209 ip-10-0-133-183 kubenswrapper[2573]: W0709 15:31:17.423701 2573 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 09 15:31:17.429209 ip-10-0-133-183 kubenswrapper[2573]: W0709 15:31:17.423703 2573 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 09 15:31:17.429209 ip-10-0-133-183 kubenswrapper[2573]: W0709 15:31:17.423705 2573 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 09 15:31:17.429209 ip-10-0-133-183 kubenswrapper[2573]: W0709 15:31:17.423708 2573 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 09 15:31:17.429209 ip-10-0-133-183 kubenswrapper[2573]: W0709 15:31:17.423710 2573 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 09 15:31:17.429209 ip-10-0-133-183 kubenswrapper[2573]: W0709 15:31:17.423713 2573 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 09 15:31:17.429209 ip-10-0-133-183 kubenswrapper[2573]: W0709 15:31:17.423715 2573 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 09 15:31:17.429209 ip-10-0-133-183 kubenswrapper[2573]: W0709 15:31:17.423718 2573 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 09 15:31:17.429209 ip-10-0-133-183 kubenswrapper[2573]: W0709 15:31:17.423720 2573 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 09 15:31:17.429209 ip-10-0-133-183 kubenswrapper[2573]: W0709 15:31:17.423723 2573 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 09 15:31:17.429209 ip-10-0-133-183 kubenswrapper[2573]: W0709 15:31:17.423725 2573 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 09 15:31:17.429209 ip-10-0-133-183 kubenswrapper[2573]: W0709 15:31:17.423728 2573 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 09 15:31:17.429209 ip-10-0-133-183 kubenswrapper[2573]: W0709 15:31:17.423730 2573 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 09 15:31:17.429209 ip-10-0-133-183 kubenswrapper[2573]: W0709 15:31:17.423733 2573 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 09 15:31:17.429209 ip-10-0-133-183 kubenswrapper[2573]: W0709 15:31:17.423735 2573 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 09 15:31:17.429685 ip-10-0-133-183 kubenswrapper[2573]: W0709 15:31:17.423738 2573 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 09 15:31:17.429685 ip-10-0-133-183 kubenswrapper[2573]: W0709 15:31:17.423740 2573 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 09 15:31:17.429685 ip-10-0-133-183 kubenswrapper[2573]: W0709 15:31:17.423743 2573 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 09 15:31:17.429685 ip-10-0-133-183 kubenswrapper[2573]: W0709 15:31:17.423745 2573 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 09 15:31:17.429685 ip-10-0-133-183 kubenswrapper[2573]: W0709 15:31:17.423747 2573 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 09 15:31:17.429685 ip-10-0-133-183 kubenswrapper[2573]: W0709 15:31:17.423751 2573 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 09 15:31:17.429685 ip-10-0-133-183 kubenswrapper[2573]: W0709 15:31:17.423755 2573 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 09 15:31:17.429685 ip-10-0-133-183 kubenswrapper[2573]: W0709 15:31:17.423757 2573 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 09 15:31:17.429685 ip-10-0-133-183 kubenswrapper[2573]: W0709 15:31:17.423760 2573 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 09 15:31:17.429685 ip-10-0-133-183 kubenswrapper[2573]: W0709 15:31:17.423764 2573 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 09 15:31:17.429685 ip-10-0-133-183 kubenswrapper[2573]: W0709 15:31:17.423766 2573 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 09 15:31:17.429685 ip-10-0-133-183 kubenswrapper[2573]: W0709 15:31:17.423769 2573 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 09 15:31:17.429685 ip-10-0-133-183 kubenswrapper[2573]: W0709 15:31:17.423772 2573 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 09 15:31:17.429685 ip-10-0-133-183 kubenswrapper[2573]: W0709 15:31:17.423774 2573 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 09 15:31:17.429685 ip-10-0-133-183 kubenswrapper[2573]: W0709 15:31:17.423777 2573 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 09 15:31:17.429685 ip-10-0-133-183 kubenswrapper[2573]: W0709 15:31:17.423779 2573 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 09 15:31:17.429685 ip-10-0-133-183 kubenswrapper[2573]: W0709 15:31:17.423782 2573 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 09 15:31:17.429685 ip-10-0-133-183 kubenswrapper[2573]: W0709 15:31:17.423785 2573 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 09 15:31:17.429685 ip-10-0-133-183 kubenswrapper[2573]: W0709 15:31:17.423787 2573 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 09 15:31:17.429685 ip-10-0-133-183 kubenswrapper[2573]: W0709 15:31:17.423790 2573 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 09 15:31:17.430170 ip-10-0-133-183 kubenswrapper[2573]: W0709 15:31:17.423792 2573 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 09 15:31:17.430170 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:31:17.423797 2573 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jul 09 15:31:17.430170 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:31:17.423925 2573 server.go:962] "Client rotation is on, will bootstrap in background" Jul 09 15:31:17.430170 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:31:17.429088 2573 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir" Jul 09 15:31:17.430657 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:31:17.430643 2573 server.go:1019] "Starting client certificate rotation" Jul 09 15:31:17.430767 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:31:17.430746 2573 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet" Jul 09 15:31:17.430796 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:31:17.430786 2573 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet" Jul 09 15:31:17.433316 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:31:17.433300 2573 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Jul 09 15:31:17.439707 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:31:17.439678 2573 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Jul 09 15:31:17.450316 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:31:17.450295 2573 log.go:25] "Validated CRI v1 runtime API" Jul 09 15:31:17.452645 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:31:17.452627 2573 log.go:25] "Validated CRI v1 image API" Jul 09 15:31:17.453550 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:31:17.453534 2573 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd" Jul 09 15:31:17.458627 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:31:17.458604 2573 fs.go:135] Filesystem UUIDs: map[1aa1b9b1-8b73-49d5-bfb1-0614bd185dd2:/dev/nvme0n1p4 7B77-95E7:/dev/nvme0n1p2 8156c19d-2946-4a99-afed-12cb670ffd04:/dev/nvme0n1p3] Jul 09 15:31:17.458695 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:31:17.458625 2573 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}] Jul 09 15:31:17.463078 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:31:17.462939 2573 manager.go:216] Machine: {Timestamp:2026-07-09 15:31:17.462455798 +0000 UTC m=+0.314865619 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3098190 MemoryCapacity:33164500992 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec202c1dfb784e5412a9a58a0c560e06 SystemUUID:ec202c1d-fb78-4e54-12a9-a58a0c560e06 BootID:fd36e21d-8d96-4a8a-8c16-3756943956ad Filesystems:[{Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16582250496 Type:vfs Inodes:4048401 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6632902656 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6107136 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16582250496 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:29:77:88:60:81 Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:29:77:88:60:81 Speed:0 Mtu:9001} {Name:ovs-system MacAddress:fa:f6:61:d3:8f:cb Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164500992 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None} Jul 09 15:31:17.463078 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:31:17.463067 2573 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available. Jul 09 15:31:17.463271 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:31:17.463156 2573 manager.go:232] Version: {KernelVersion:5.14.0-570.126.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260702-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:} Jul 09 15:31:17.464182 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:31:17.464164 2573 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Jul 09 15:31:17.466199 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:31:17.466167 2573 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[] Jul 09 15:31:17.466356 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:31:17.466202 2573 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-133-183.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2} Jul 09 15:31:17.467445 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:31:17.467434 2573 topology_manager.go:138] "Creating topology manager with none policy" Jul 09 15:31:17.467477 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:31:17.467447 2573 container_manager_linux.go:309] "Creating device plugin manager" Jul 09 15:31:17.467477 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:31:17.467455 2573 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock" Jul 09 15:31:17.468008 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:31:17.467998 2573 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock" Jul 09 15:31:17.469133 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:31:17.469123 2573 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager" Jul 09 15:31:17.472380 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:31:17.472367 2573 state_mem.go:36] "Initialized new in-memory state store" Jul 09 15:31:17.472570 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:31:17.472552 2573 server.go:1240] "Using root directory" path="/var/lib/kubelet" Jul 09 15:31:17.472613 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:31:17.472595 2573 kubelet.go:486] "Attempting to sync node with API server" Jul 09 15:31:17.472641 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:31:17.472615 2573 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests" Jul 09 15:31:17.472641 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:31:17.472632 2573 file.go:69] "Watching path" path="/etc/kubernetes/manifests" Jul 09 15:31:17.472707 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:31:17.472643 2573 kubelet.go:398] "Adding apiserver pod source" Jul 09 15:31:17.472707 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:31:17.472655 2573 apiserver.go:42] "Waiting for node sync before watching apiserver pods" Jul 09 15:31:17.473168 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:31:17.473156 2573 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Jul 09 15:31:17.473209 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:31:17.473176 2573 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Jul 09 15:31:17.476633 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:31:17.476604 2573 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.10-2.rhaos4.21.git83ee25b.el9" apiVersion="v1" Jul 09 15:31:17.478455 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:31:17.478440 2573 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled" Jul 09 15:31:17.478502 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:31:17.478471 2573 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled" Jul 09 15:31:17.479562 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:31:17.479550 2573 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume" Jul 09 15:31:17.479597 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:31:17.479567 2573 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir" Jul 09 15:31:17.479597 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:31:17.479574 2573 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo" Jul 09 15:31:17.479597 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:31:17.479579 2573 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path" Jul 09 15:31:17.479597 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:31:17.479587 2573 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs" Jul 09 15:31:17.479597 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:31:17.479593 2573 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret" Jul 09 15:31:17.479734 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:31:17.479600 2573 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi" Jul 09 15:31:17.479734 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:31:17.479606 2573 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api" Jul 09 15:31:17.479734 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:31:17.479613 2573 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc" Jul 09 15:31:17.479734 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:31:17.479619 2573 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap" Jul 09 15:31:17.479734 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:31:17.479627 2573 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected" Jul 09 15:31:17.479734 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:31:17.479635 2573 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume" Jul 09 15:31:17.480528 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:31:17.480492 2573 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi" Jul 09 15:31:17.480680 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:31:17.480658 2573 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image" Jul 09 15:31:17.483470 ip-10-0-133-183 kubenswrapper[2573]: E0709 15:31:17.483445 2573 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-133-183.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node" Jul 09 15:31:17.483556 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:31:17.483472 2573 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-133-183.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Jul 09 15:31:17.483556 ip-10-0-133-183 kubenswrapper[2573]: E0709 15:31:17.483440 2573 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service" Jul 09 15:31:17.484175 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:31:17.484155 2573 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-2lzvv" Jul 09 15:31:17.484679 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:31:17.484665 2573 server.go:1268] "Started kubelet" Jul 09 15:31:17.484809 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:31:17.484747 2573 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10 Jul 09 15:31:17.484931 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:31:17.484831 2573 server_v1.go:49] "podresources" method="list" useActivePods=true Jul 09 15:31:17.485400 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:31:17.485371 2573 server.go:180] "Starting to listen" address="0.0.0.0" port=10250 Jul 09 15:31:17.485549 ip-10-0-133-183 systemd[1]: Started Kubernetes Kubelet. Jul 09 15:31:17.486213 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:31:17.486186 2573 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock" Jul 09 15:31:17.487307 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:31:17.487290 2573 server.go:310] "Adding debug handlers to kubelet server" Jul 09 15:31:17.489523 ip-10-0-133-183 kubenswrapper[2573]: E0709 15:31:17.488393 2573 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-133-183.ec2.internal.18c0a8d332ec5199 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-133-183.ec2.internal,UID:ip-10-0-133-183.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-133-183.ec2.internal,},FirstTimestamp:2026-07-09 15:31:17.484630425 +0000 UTC m=+0.337040247,LastTimestamp:2026-07-09 15:31:17.484630425 +0000 UTC m=+0.337040247,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-133-183.ec2.internal,}" Jul 09 15:31:17.491487 ip-10-0-133-183 kubenswrapper[2573]: E0709 15:31:17.491467 2573 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache" Jul 09 15:31:17.492149 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:31:17.492135 2573 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer" Jul 09 15:31:17.492220 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:31:17.492200 2573 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving" Jul 09 15:31:17.492376 ip-10-0-133-183 kubenswrapper[2573]: E0709 15:31:17.492353 2573 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-133-183.ec2.internal\" not found" Jul 09 15:31:17.492724 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:31:17.492694 2573 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-2lzvv" Jul 09 15:31:17.493143 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:31:17.492937 2573 volume_manager.go:311] "The desired_state_of_world populator starts" Jul 09 15:31:17.493143 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:31:17.492958 2573 volume_manager.go:313] "Starting Kubelet Volume Manager" Jul 09 15:31:17.493143 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:31:17.492942 2573 desired_state_of_world_populator.go:146] "Desired state populator starts to run" Jul 09 15:31:17.493143 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:31:17.493096 2573 reconstruct.go:97] "Volume reconstruction finished" Jul 09 15:31:17.493143 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:31:17.493104 2573 reconciler.go:29] "Reconciler: start to sync state" Jul 09 15:31:17.493342 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:31:17.493204 2573 factory.go:55] Registering systemd factory Jul 09 15:31:17.493342 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:31:17.493221 2573 factory.go:223] Registration of the systemd container factory successfully Jul 09 15:31:17.493475 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:31:17.493457 2573 factory.go:153] Registering CRI-O factory Jul 09 15:31:17.493541 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:31:17.493478 2573 factory.go:223] Registration of the crio container factory successfully Jul 09 15:31:17.493575 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:31:17.493543 2573 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory Jul 09 15:31:17.493575 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:31:17.493570 2573 factory.go:103] Registering Raw factory Jul 09 15:31:17.493674 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:31:17.493616 2573 manager.go:1196] Started watching for new ooms in manager Jul 09 15:31:17.494400 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:31:17.494383 2573 manager.go:349] Starting recovery of all containers Jul 09 15:31:17.504867 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:31:17.504834 2573 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160" Jul 09 15:31:17.507086 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:31:17.507063 2573 manager.go:354] Recovery completed Jul 09 15:31:17.510045 ip-10-0-133-183 kubenswrapper[2573]: E0709 15:31:17.509812 2573 nodelease.go:49] "Failed to get node when trying to set owner ref to the node lease" err="nodes \"ip-10-0-133-183.ec2.internal\" not found" node="ip-10-0-133-183.ec2.internal" Jul 09 15:31:17.512655 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:31:17.512637 2573 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 09 15:31:17.514494 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:31:17.514474 2573 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-183.ec2.internal" event="NodeHasSufficientMemory" Jul 09 15:31:17.514579 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:31:17.514506 2573 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-183.ec2.internal" event="NodeHasNoDiskPressure" Jul 09 15:31:17.514579 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:31:17.514522 2573 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-183.ec2.internal" event="NodeHasSufficientPID" Jul 09 15:31:17.515053 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:31:17.515033 2573 cpu_manager.go:222] "Starting CPU manager" policy="none" Jul 09 15:31:17.515053 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:31:17.515048 2573 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 09 15:31:17.515053 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:31:17.515051 2573 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s" Jul 09 15:31:17.515341 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:31:17.515071 2573 state_mem.go:36] "Initialized new in-memory state store" Jul 09 15:31:17.515781 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:31:17.515766 2573 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-183.ec2.internal" event="NodeHasSufficientMemory" Jul 09 15:31:17.515849 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:31:17.515800 2573 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-183.ec2.internal" event="NodeHasNoDiskPressure" Jul 09 15:31:17.515849 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:31:17.515815 2573 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-183.ec2.internal" event="NodeHasSufficientPID" Jul 09 15:31:17.516206 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:31:17.516193 2573 policy_none.go:49] "None policy: Start" Jul 09 15:31:17.516270 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:31:17.516209 2573 memory_manager.go:187] "Starting memorymanager" policy="None" Jul 09 15:31:17.516270 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:31:17.516221 2573 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint" Jul 09 15:31:17.517280 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:31:17.517265 2573 policy_none.go:47] "Start" Jul 09 15:31:17.562597 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:31:17.562576 2573 manager.go:337] "Starting Device Plugin manager" Jul 09 15:31:17.562751 ip-10-0-133-183 kubenswrapper[2573]: E0709 15:31:17.562614 2573 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint" Jul 09 15:31:17.562751 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:31:17.562625 2573 server.go:87] "Starting device plugin registration server" Jul 09 15:31:17.562876 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:31:17.562864 2573 eviction_manager.go:189] "Eviction manager: starting control loop" Jul 09 15:31:17.562926 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:31:17.562878 2573 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s" Jul 09 15:31:17.563015 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:31:17.563001 2573 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry" Jul 09 15:31:17.563090 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:31:17.563079 2573 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts" Jul 09 15:31:17.563135 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:31:17.563091 2573 plugin_manager.go:118] "Starting Kubelet Plugin Manager" Jul 09 15:31:17.563693 ip-10-0-133-183 kubenswrapper[2573]: E0709 15:31:17.563672 2573 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\"" Jul 09 15:31:17.563778 ip-10-0-133-183 kubenswrapper[2573]: E0709 15:31:17.563732 2573 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-133-183.ec2.internal\" not found" Jul 09 15:31:17.663590 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:31:17.663478 2573 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 09 15:31:17.664750 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:31:17.664561 2573 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-183.ec2.internal" event="NodeHasSufficientMemory" Jul 09 15:31:17.664750 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:31:17.664591 2573 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-183.ec2.internal" event="NodeHasNoDiskPressure" Jul 09 15:31:17.664750 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:31:17.664601 2573 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-183.ec2.internal" event="NodeHasSufficientPID" Jul 09 15:31:17.664750 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:31:17.664632 2573 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-133-183.ec2.internal" Jul 09 15:31:17.672234 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:31:17.672208 2573 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4" Jul 09 15:31:17.673565 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:31:17.673545 2573 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6" Jul 09 15:31:17.673683 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:31:17.673591 2573 status_manager.go:244] "Starting to sync pod status with apiserver" Jul 09 15:31:17.673683 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:31:17.673616 2573 kubelet.go:2448] "Starting kubelet main sync loop" Jul 09 15:31:17.673683 ip-10-0-133-183 kubenswrapper[2573]: E0709 15:31:17.673652 2573 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful" Jul 09 15:31:17.674202 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:31:17.674185 2573 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-133-183.ec2.internal" Jul 09 15:31:17.674310 ip-10-0-133-183 kubenswrapper[2573]: E0709 15:31:17.674212 2573 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-133-183.ec2.internal\": node \"ip-10-0-133-183.ec2.internal\" not found" Jul 09 15:31:17.676022 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:31:17.676002 2573 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160" Jul 09 15:31:17.705925 ip-10-0-133-183 kubenswrapper[2573]: E0709 15:31:17.705895 2573 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-133-183.ec2.internal\" not found" Jul 09 15:31:17.774146 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:31:17.774104 2573 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["kube-system/kube-apiserver-proxy-ip-10-0-133-183.ec2.internal","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-183.ec2.internal"] Jul 09 15:31:17.774230 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:31:17.774206 2573 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 09 15:31:17.775913 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:31:17.775894 2573 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-183.ec2.internal" event="NodeHasSufficientMemory" Jul 09 15:31:17.776034 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:31:17.775926 2573 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-183.ec2.internal" event="NodeHasNoDiskPressure" Jul 09 15:31:17.776034 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:31:17.775936 2573 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-183.ec2.internal" event="NodeHasSufficientPID" Jul 09 15:31:17.777192 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:31:17.777179 2573 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 09 15:31:17.777289 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:31:17.777273 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-133-183.ec2.internal" Jul 09 15:31:17.777346 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:31:17.777304 2573 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 09 15:31:17.777905 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:31:17.777886 2573 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-183.ec2.internal" event="NodeHasSufficientMemory" Jul 09 15:31:17.778015 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:31:17.777917 2573 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-183.ec2.internal" event="NodeHasNoDiskPressure" Jul 09 15:31:17.778015 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:31:17.777928 2573 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-183.ec2.internal" event="NodeHasSufficientPID" Jul 09 15:31:17.778015 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:31:17.777886 2573 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-183.ec2.internal" event="NodeHasSufficientMemory" Jul 09 15:31:17.778015 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:31:17.778005 2573 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-183.ec2.internal" event="NodeHasNoDiskPressure" Jul 09 15:31:17.778147 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:31:17.778018 2573 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-183.ec2.internal" event="NodeHasSufficientPID" Jul 09 15:31:17.779130 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:31:17.779109 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-183.ec2.internal" Jul 09 15:31:17.779176 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:31:17.779142 2573 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 09 15:31:17.779891 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:31:17.779876 2573 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-183.ec2.internal" event="NodeHasSufficientMemory" Jul 09 15:31:17.779959 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:31:17.779902 2573 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-183.ec2.internal" event="NodeHasNoDiskPressure" Jul 09 15:31:17.779959 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:31:17.779915 2573 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-183.ec2.internal" event="NodeHasSufficientPID" Jul 09 15:31:17.795045 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:31:17.795023 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/fe0ed806b6a4ba1e258c73e86f255df7-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-133-183.ec2.internal\" (UID: \"fe0ed806b6a4ba1e258c73e86f255df7\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-183.ec2.internal" Jul 09 15:31:17.795127 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:31:17.795049 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/fe0ed806b6a4ba1e258c73e86f255df7-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-133-183.ec2.internal\" (UID: \"fe0ed806b6a4ba1e258c73e86f255df7\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-183.ec2.internal" Jul 09 15:31:17.795127 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:31:17.795067 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/06227558853f1ac07df9b5c4930c43af-config\") pod \"kube-apiserver-proxy-ip-10-0-133-183.ec2.internal\" (UID: \"06227558853f1ac07df9b5c4930c43af\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-133-183.ec2.internal" Jul 09 15:31:17.800683 ip-10-0-133-183 kubenswrapper[2573]: E0709 15:31:17.800661 2573 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-133-183.ec2.internal\" not found" node="ip-10-0-133-183.ec2.internal" Jul 09 15:31:17.805090 ip-10-0-133-183 kubenswrapper[2573]: E0709 15:31:17.805075 2573 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-133-183.ec2.internal\" not found" node="ip-10-0-133-183.ec2.internal" Jul 09 15:31:17.806017 ip-10-0-133-183 kubenswrapper[2573]: E0709 15:31:17.806002 2573 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-133-183.ec2.internal\" not found" Jul 09 15:31:17.895991 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:31:17.895940 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/06227558853f1ac07df9b5c4930c43af-config\") pod \"kube-apiserver-proxy-ip-10-0-133-183.ec2.internal\" (UID: \"06227558853f1ac07df9b5c4930c43af\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-133-183.ec2.internal" Jul 09 15:31:17.895991 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:31:17.895991 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/fe0ed806b6a4ba1e258c73e86f255df7-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-133-183.ec2.internal\" (UID: \"fe0ed806b6a4ba1e258c73e86f255df7\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-183.ec2.internal" Jul 09 15:31:17.896170 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:31:17.896010 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/fe0ed806b6a4ba1e258c73e86f255df7-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-133-183.ec2.internal\" (UID: \"fe0ed806b6a4ba1e258c73e86f255df7\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-183.ec2.internal" Jul 09 15:31:17.896170 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:31:17.896040 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/06227558853f1ac07df9b5c4930c43af-config\") pod \"kube-apiserver-proxy-ip-10-0-133-183.ec2.internal\" (UID: \"06227558853f1ac07df9b5c4930c43af\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-133-183.ec2.internal" Jul 09 15:31:17.896170 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:31:17.896044 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/fe0ed806b6a4ba1e258c73e86f255df7-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-133-183.ec2.internal\" (UID: \"fe0ed806b6a4ba1e258c73e86f255df7\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-183.ec2.internal" Jul 09 15:31:17.896170 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:31:17.896046 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/fe0ed806b6a4ba1e258c73e86f255df7-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-133-183.ec2.internal\" (UID: \"fe0ed806b6a4ba1e258c73e86f255df7\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-183.ec2.internal" Jul 09 15:31:17.906647 ip-10-0-133-183 kubenswrapper[2573]: E0709 15:31:17.906621 2573 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-133-183.ec2.internal\" not found" Jul 09 15:31:18.007728 ip-10-0-133-183 kubenswrapper[2573]: E0709 15:31:18.007650 2573 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-133-183.ec2.internal\" not found" Jul 09 15:31:18.104260 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:31:18.104230 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-133-183.ec2.internal" Jul 09 15:31:18.108727 ip-10-0-133-183 kubenswrapper[2573]: E0709 15:31:18.108507 2573 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-133-183.ec2.internal\" not found" Jul 09 15:31:18.109632 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:31:18.109616 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-183.ec2.internal" Jul 09 15:31:18.208663 ip-10-0-133-183 kubenswrapper[2573]: E0709 15:31:18.208621 2573 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-133-183.ec2.internal\" not found" Jul 09 15:31:18.309241 ip-10-0-133-183 kubenswrapper[2573]: E0709 15:31:18.309182 2573 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-133-183.ec2.internal\" not found" Jul 09 15:31:18.409727 ip-10-0-133-183 kubenswrapper[2573]: E0709 15:31:18.409695 2573 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-133-183.ec2.internal\" not found" Jul 09 15:31:18.429891 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:31:18.429857 2573 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials" Jul 09 15:31:18.430058 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:31:18.430040 2573 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received" Jul 09 15:31:18.430099 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:31:18.430059 2573 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received" Jul 09 15:31:18.497655 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:31:18.497622 2573 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-07-08 15:26:18 +0000 UTC" deadline="2027-12-15 19:21:41.273771385 +0000 UTC" Jul 09 15:31:18.497655 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:31:18.497651 2573 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="12579h50m22.776123244s" Jul 09 15:31:18.510699 ip-10-0-133-183 kubenswrapper[2573]: E0709 15:31:18.510677 2573 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-133-183.ec2.internal\" not found" Jul 09 15:31:18.611397 ip-10-0-133-183 kubenswrapper[2573]: E0709 15:31:18.611323 2573 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-133-183.ec2.internal\" not found" Jul 09 15:31:18.638208 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:31:18.638182 2573 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160" Jul 09 15:31:18.711668 ip-10-0-133-183 kubenswrapper[2573]: E0709 15:31:18.711631 2573 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-133-183.ec2.internal\" not found" Jul 09 15:31:18.732326 ip-10-0-133-183 kubenswrapper[2573]: W0709 15:31:18.732286 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod06227558853f1ac07df9b5c4930c43af.slice/crio-324e0c6e153c67ec232015060c7a16578a083e3d3f6c3abba6ab7675b562b0e9 WatchSource:0}: Error finding container 324e0c6e153c67ec232015060c7a16578a083e3d3f6c3abba6ab7675b562b0e9: Status 404 returned error can't find the container with id 324e0c6e153c67ec232015060c7a16578a083e3d3f6c3abba6ab7675b562b0e9 Jul 09 15:31:18.737236 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:31:18.737220 2573 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 09 15:31:18.812476 ip-10-0-133-183 kubenswrapper[2573]: E0709 15:31:18.812442 2573 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-133-183.ec2.internal\" not found" Jul 09 15:31:18.913055 ip-10-0-133-183 kubenswrapper[2573]: E0709 15:31:18.912987 2573 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-133-183.ec2.internal\" not found" Jul 09 15:31:19.013882 ip-10-0-133-183 kubenswrapper[2573]: E0709 15:31:19.013839 2573 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-133-183.ec2.internal\" not found" Jul 09 15:31:19.020887 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:31:19.020868 2573 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160" Jul 09 15:31:19.093375 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:31:19.093340 2573 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-133-183.ec2.internal" Jul 09 15:31:20.236237 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:31:20.236220 2573 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Jul 09 15:31:20.240168 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:31:20.240137 2573 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-183.ec2.internal" Jul 09 15:31:20.253720 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:31:20.253698 2573 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Jul 09 15:31:20.605189 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:31:20.605155 2573 apiserver.go:52] "Watching apiserver" Jul 09 15:31:20.615890 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:31:20.615855 2573 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66" Jul 09 15:31:20.616788 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:31:20.616768 2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-multus/network-metrics-daemon-8f2p4","kube-system/konnectivity-agent-zrrxr","kube-system/kube-apiserver-proxy-ip-10-0-133-183.ec2.internal","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vxr6x","openshift-network-diagnostics/network-check-target-sdbbl","openshift-network-operator/iptables-alerter-7zxwk","openshift-ovn-kubernetes/ovnkube-node-b7b9v","kube-system/global-pull-secret-syncer-h2cs6","openshift-cluster-node-tuning-operator/tuned-zp45w","openshift-dns/node-resolver-b5c8h","openshift-image-registry/node-ca-trnjd","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-183.ec2.internal","openshift-multus/multus-additional-cni-plugins-g5qvc","openshift-multus/multus-v27h9"] Jul 09 15:31:20.619200 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:31:20.619180 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-b7b9v" Jul 09 15:31:20.620377 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:31:20.620360 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-g5qvc" Jul 09 15:31:20.621586 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:31:20.621568 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-v27h9" Jul 09 15:31:20.622059 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:31:20.622041 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\"" Jul 09 15:31:20.622148 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:31:20.622093 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\"" Jul 09 15:31:20.622276 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:31:20.622263 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\"" Jul 09 15:31:20.622326 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:31:20.622294 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-pk4qz\"" Jul 09 15:31:20.622396 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:31:20.622379 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\"" Jul 09 15:31:20.622744 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:31:20.622729 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vxr6x" Jul 09 15:31:20.623792 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:31:20.623768 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\"" Jul 09 15:31:20.623792 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:31:20.623775 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\"" Jul 09 15:31:20.623792 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:31:20.623775 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\"" Jul 09 15:31:20.624078 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:31:20.623819 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\"" Jul 09 15:31:20.624078 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:31:20.623890 2573 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving" Jul 09 15:31:20.624078 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:31:20.623910 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\"" Jul 09 15:31:20.624078 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:31:20.623984 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\"" Jul 09 15:31:20.624078 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:31:20.624037 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\"" Jul 09 15:31:20.624303 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:31:20.624229 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-sdbbl" Jul 09 15:31:20.624357 ip-10-0-133-183 kubenswrapper[2573]: E0709 15:31:20.624299 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-sdbbl" podUID="e8de13cc-b0e0-4a54-bd44-3ba73cbf2bf2" Jul 09 15:31:20.624662 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:31:20.624645 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-qmmn6\"" Jul 09 15:31:20.625227 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:31:20.625209 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\"" Jul 09 15:31:20.625316 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:31:20.625230 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-w8mtk\"" Jul 09 15:31:20.625374 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:31:20.625348 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-7zxwk" Jul 09 15:31:20.625446 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:31:20.625430 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\"" Jul 09 15:31:20.625497 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:31:20.625465 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-njzql\"" Jul 09 15:31:20.625704 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:31:20.625688 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\"" Jul 09 15:31:20.625761 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:31:20.625723 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\"" Jul 09 15:31:20.628993 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:31:20.627540 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-zp45w" Jul 09 15:31:20.628993 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:31:20.628898 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-8f2p4" Jul 09 15:31:20.628993 ip-10-0-133-183 kubenswrapper[2573]: E0709 15:31:20.628973 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-8f2p4" podUID="8e56f4e6-cbec-422f-9505-c7c6654159d9" Jul 09 15:31:20.629858 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:31:20.629444 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-l8cv6\"" Jul 09 15:31:20.629858 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:31:20.629702 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\"" Jul 09 15:31:20.630061 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:31:20.630026 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\"" Jul 09 15:31:20.630136 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:31:20.630041 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\"" Jul 09 15:31:20.630185 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:31:20.630170 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\"" Jul 09 15:31:20.630282 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:31:20.630247 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-tbb8h\"" Jul 09 15:31:20.630452 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:31:20.630433 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\"" Jul 09 15:31:20.631298 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:31:20.631279 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-zrrxr" Jul 09 15:31:20.632368 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:31:20.632349 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-b5c8h" Jul 09 15:31:20.633771 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:31:20.633752 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-trnjd" Jul 09 15:31:20.634152 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:31:20.634024 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\"" Jul 09 15:31:20.634152 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:31:20.634050 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\"" Jul 09 15:31:20.634152 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:31:20.634031 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-5jh8v\"" Jul 09 15:31:20.635066 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:31:20.635052 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\"" Jul 09 15:31:20.635142 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:31:20.635065 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\"" Jul 09 15:31:20.635142 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:31:20.635077 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-h2cs6" Jul 09 15:31:20.635142 ip-10-0-133-183 kubenswrapper[2573]: E0709 15:31:20.635128 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-h2cs6" podUID="d9ea5934-356a-47e5-9fc6-1ae4f39f4e19" Jul 09 15:31:20.635282 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:31:20.635269 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-xfpjk\"" Jul 09 15:31:20.636155 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:31:20.636139 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/db22f88d-a953-488e-b384-67f991351f73-host-slash\") pod \"ovnkube-node-b7b9v\" (UID: \"db22f88d-a953-488e-b384-67f991351f73\") " pod="openshift-ovn-kubernetes/ovnkube-node-b7b9v" Jul 09 15:31:20.636229 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:31:20.636163 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/db22f88d-a953-488e-b384-67f991351f73-ovnkube-script-lib\") pod \"ovnkube-node-b7b9v\" (UID: \"db22f88d-a953-488e-b384-67f991351f73\") " pod="openshift-ovn-kubernetes/ovnkube-node-b7b9v" Jul 09 15:31:20.636229 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:31:20.636180 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/86ea7a2d-fc63-4660-a266-93fe5360fe49-system-cni-dir\") pod \"multus-v27h9\" (UID: \"86ea7a2d-fc63-4660-a266-93fe5360fe49\") " pod="openshift-multus/multus-v27h9" Jul 09 15:31:20.636229 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:31:20.636201 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/86ea7a2d-fc63-4660-a266-93fe5360fe49-host-run-k8s-cni-cncf-io\") pod \"multus-v27h9\" (UID: \"86ea7a2d-fc63-4660-a266-93fe5360fe49\") " pod="openshift-multus/multus-v27h9" Jul 09 15:31:20.636365 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:31:20.636252 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/db22f88d-a953-488e-b384-67f991351f73-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-b7b9v\" (UID: \"db22f88d-a953-488e-b384-67f991351f73\") " pod="openshift-ovn-kubernetes/ovnkube-node-b7b9v" Jul 09 15:31:20.636365 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:31:20.636306 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/86ea7a2d-fc63-4660-a266-93fe5360fe49-etc-kubernetes\") pod \"multus-v27h9\" (UID: \"86ea7a2d-fc63-4660-a266-93fe5360fe49\") " pod="openshift-multus/multus-v27h9" Jul 09 15:31:20.636365 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:31:20.636352 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/db22f88d-a953-488e-b384-67f991351f73-host-kubelet\") pod \"ovnkube-node-b7b9v\" (UID: \"db22f88d-a953-488e-b384-67f991351f73\") " pod="openshift-ovn-kubernetes/ovnkube-node-b7b9v" Jul 09 15:31:20.636505 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:31:20.636375 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/db22f88d-a953-488e-b384-67f991351f73-host-cni-bin\") pod \"ovnkube-node-b7b9v\" (UID: \"db22f88d-a953-488e-b384-67f991351f73\") " pod="openshift-ovn-kubernetes/ovnkube-node-b7b9v" Jul 09 15:31:20.636505 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:31:20.636393 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/85842275-e9c5-44af-8e99-1700a37602e3-system-cni-dir\") pod \"multus-additional-cni-plugins-g5qvc\" (UID: \"85842275-e9c5-44af-8e99-1700a37602e3\") " pod="openshift-multus/multus-additional-cni-plugins-g5qvc" Jul 09 15:31:20.636505 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:31:20.636442 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-df9gg\" (UniqueName: \"kubernetes.io/projected/86ea7a2d-fc63-4660-a266-93fe5360fe49-kube-api-access-df9gg\") pod \"multus-v27h9\" (UID: \"86ea7a2d-fc63-4660-a266-93fe5360fe49\") " pod="openshift-multus/multus-v27h9" Jul 09 15:31:20.636505 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:31:20.636476 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/38e3cf1c-0139-47f3-a0f2-708c4e43b855-device-dir\") pod \"aws-ebs-csi-driver-node-vxr6x\" (UID: \"38e3cf1c-0139-47f3-a0f2-708c4e43b855\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vxr6x" Jul 09 15:31:20.636691 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:31:20.636520 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/85842275-e9c5-44af-8e99-1700a37602e3-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-g5qvc\" (UID: \"85842275-e9c5-44af-8e99-1700a37602e3\") " pod="openshift-multus/multus-additional-cni-plugins-g5qvc" Jul 09 15:31:20.636691 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:31:20.636553 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/86ea7a2d-fc63-4660-a266-93fe5360fe49-multus-cni-dir\") pod \"multus-v27h9\" (UID: \"86ea7a2d-fc63-4660-a266-93fe5360fe49\") " pod="openshift-multus/multus-v27h9" Jul 09 15:31:20.636691 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:31:20.636561 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\"" Jul 09 15:31:20.636691 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:31:20.636592 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/86ea7a2d-fc63-4660-a266-93fe5360fe49-host-var-lib-kubelet\") pod \"multus-v27h9\" (UID: \"86ea7a2d-fc63-4660-a266-93fe5360fe49\") " pod="openshift-multus/multus-v27h9" Jul 09 15:31:20.636691 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:31:20.636632 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/38e3cf1c-0139-47f3-a0f2-708c4e43b855-socket-dir\") pod \"aws-ebs-csi-driver-node-vxr6x\" (UID: \"38e3cf1c-0139-47f3-a0f2-708c4e43b855\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vxr6x" Jul 09 15:31:20.636691 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:31:20.636658 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/db22f88d-a953-488e-b384-67f991351f73-etc-openvswitch\") pod \"ovnkube-node-b7b9v\" (UID: \"db22f88d-a953-488e-b384-67f991351f73\") " pod="openshift-ovn-kubernetes/ovnkube-node-b7b9v" Jul 09 15:31:20.636691 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:31:20.636683 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/db22f88d-a953-488e-b384-67f991351f73-ovnkube-config\") pod \"ovnkube-node-b7b9v\" (UID: \"db22f88d-a953-488e-b384-67f991351f73\") " pod="openshift-ovn-kubernetes/ovnkube-node-b7b9v" Jul 09 15:31:20.637018 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:31:20.636740 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/86ea7a2d-fc63-4660-a266-93fe5360fe49-multus-socket-dir-parent\") pod \"multus-v27h9\" (UID: \"86ea7a2d-fc63-4660-a266-93fe5360fe49\") " pod="openshift-multus/multus-v27h9" Jul 09 15:31:20.637018 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:31:20.636766 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/86ea7a2d-fc63-4660-a266-93fe5360fe49-host-run-multus-certs\") pod \"multus-v27h9\" (UID: \"86ea7a2d-fc63-4660-a266-93fe5360fe49\") " pod="openshift-multus/multus-v27h9" Jul 09 15:31:20.637018 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:31:20.636789 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6lttq\" (UniqueName: \"kubernetes.io/projected/38e3cf1c-0139-47f3-a0f2-708c4e43b855-kube-api-access-6lttq\") pod \"aws-ebs-csi-driver-node-vxr6x\" (UID: \"38e3cf1c-0139-47f3-a0f2-708c4e43b855\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vxr6x" Jul 09 15:31:20.637018 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:31:20.636806 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/db22f88d-a953-488e-b384-67f991351f73-env-overrides\") pod \"ovnkube-node-b7b9v\" (UID: \"db22f88d-a953-488e-b384-67f991351f73\") " pod="openshift-ovn-kubernetes/ovnkube-node-b7b9v" Jul 09 15:31:20.637018 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:31:20.636821 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/85842275-e9c5-44af-8e99-1700a37602e3-cni-binary-copy\") pod \"multus-additional-cni-plugins-g5qvc\" (UID: \"85842275-e9c5-44af-8e99-1700a37602e3\") " pod="openshift-multus/multus-additional-cni-plugins-g5qvc" Jul 09 15:31:20.637018 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:31:20.636838 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/85842275-e9c5-44af-8e99-1700a37602e3-tuning-conf-dir\") pod \"multus-additional-cni-plugins-g5qvc\" (UID: \"85842275-e9c5-44af-8e99-1700a37602e3\") " pod="openshift-multus/multus-additional-cni-plugins-g5qvc" Jul 09 15:31:20.637018 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:31:20.636841 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\"" Jul 09 15:31:20.637018 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:31:20.636859 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/86ea7a2d-fc63-4660-a266-93fe5360fe49-host-run-netns\") pod \"multus-v27h9\" (UID: \"86ea7a2d-fc63-4660-a266-93fe5360fe49\") " pod="openshift-multus/multus-v27h9" Jul 09 15:31:20.637018 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:31:20.636896 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/86ea7a2d-fc63-4660-a266-93fe5360fe49-host-var-lib-cni-multus\") pod \"multus-v27h9\" (UID: \"86ea7a2d-fc63-4660-a266-93fe5360fe49\") " pod="openshift-multus/multus-v27h9" Jul 09 15:31:20.637018 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:31:20.636915 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-wl7hn\"" Jul 09 15:31:20.637018 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:31:20.636918 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/38e3cf1c-0139-47f3-a0f2-708c4e43b855-sys-fs\") pod \"aws-ebs-csi-driver-node-vxr6x\" (UID: \"38e3cf1c-0139-47f3-a0f2-708c4e43b855\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vxr6x" Jul 09 15:31:20.637018 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:31:20.636961 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/07a170e7-eeff-4591-9a10-ecfe6a8e8100-host-slash\") pod \"iptables-alerter-7zxwk\" (UID: \"07a170e7-eeff-4591-9a10-ecfe6a8e8100\") " pod="openshift-network-operator/iptables-alerter-7zxwk" Jul 09 15:31:20.637018 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:31:20.636979 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/db22f88d-a953-488e-b384-67f991351f73-ovn-node-metrics-cert\") pod \"ovnkube-node-b7b9v\" (UID: \"db22f88d-a953-488e-b384-67f991351f73\") " pod="openshift-ovn-kubernetes/ovnkube-node-b7b9v" Jul 09 15:31:20.637018 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:31:20.637012 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-njbgg\" (UniqueName: \"kubernetes.io/projected/db22f88d-a953-488e-b384-67f991351f73-kube-api-access-njbgg\") pod \"ovnkube-node-b7b9v\" (UID: \"db22f88d-a953-488e-b384-67f991351f73\") " pod="openshift-ovn-kubernetes/ovnkube-node-b7b9v" Jul 09 15:31:20.637577 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:31:20.637038 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/85842275-e9c5-44af-8e99-1700a37602e3-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-g5qvc\" (UID: \"85842275-e9c5-44af-8e99-1700a37602e3\") " pod="openshift-multus/multus-additional-cni-plugins-g5qvc" Jul 09 15:31:20.637577 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:31:20.637057 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ql4w6\" (UniqueName: \"kubernetes.io/projected/85842275-e9c5-44af-8e99-1700a37602e3-kube-api-access-ql4w6\") pod \"multus-additional-cni-plugins-g5qvc\" (UID: \"85842275-e9c5-44af-8e99-1700a37602e3\") " pod="openshift-multus/multus-additional-cni-plugins-g5qvc" Jul 09 15:31:20.637577 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:31:20.637078 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/86ea7a2d-fc63-4660-a266-93fe5360fe49-cni-binary-copy\") pod \"multus-v27h9\" (UID: \"86ea7a2d-fc63-4660-a266-93fe5360fe49\") " pod="openshift-multus/multus-v27h9" Jul 09 15:31:20.637577 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:31:20.637092 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\"" Jul 09 15:31:20.637577 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:31:20.637099 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/38e3cf1c-0139-47f3-a0f2-708c4e43b855-etc-selinux\") pod \"aws-ebs-csi-driver-node-vxr6x\" (UID: \"38e3cf1c-0139-47f3-a0f2-708c4e43b855\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vxr6x" Jul 09 15:31:20.637577 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:31:20.637131 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/db22f88d-a953-488e-b384-67f991351f73-systemd-units\") pod \"ovnkube-node-b7b9v\" (UID: \"db22f88d-a953-488e-b384-67f991351f73\") " pod="openshift-ovn-kubernetes/ovnkube-node-b7b9v" Jul 09 15:31:20.637577 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:31:20.637146 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/db22f88d-a953-488e-b384-67f991351f73-run-systemd\") pod \"ovnkube-node-b7b9v\" (UID: \"db22f88d-a953-488e-b384-67f991351f73\") " pod="openshift-ovn-kubernetes/ovnkube-node-b7b9v" Jul 09 15:31:20.637577 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:31:20.637160 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/db22f88d-a953-488e-b384-67f991351f73-node-log\") pod \"ovnkube-node-b7b9v\" (UID: \"db22f88d-a953-488e-b384-67f991351f73\") " pod="openshift-ovn-kubernetes/ovnkube-node-b7b9v" Jul 09 15:31:20.637577 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:31:20.637175 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/85842275-e9c5-44af-8e99-1700a37602e3-os-release\") pod \"multus-additional-cni-plugins-g5qvc\" (UID: \"85842275-e9c5-44af-8e99-1700a37602e3\") " pod="openshift-multus/multus-additional-cni-plugins-g5qvc" Jul 09 15:31:20.637577 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:31:20.637199 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/db22f88d-a953-488e-b384-67f991351f73-host-run-netns\") pod \"ovnkube-node-b7b9v\" (UID: \"db22f88d-a953-488e-b384-67f991351f73\") " pod="openshift-ovn-kubernetes/ovnkube-node-b7b9v" Jul 09 15:31:20.637577 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:31:20.637218 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/86ea7a2d-fc63-4660-a266-93fe5360fe49-cnibin\") pod \"multus-v27h9\" (UID: \"86ea7a2d-fc63-4660-a266-93fe5360fe49\") " pod="openshift-multus/multus-v27h9" Jul 09 15:31:20.637577 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:31:20.637234 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/db22f88d-a953-488e-b384-67f991351f73-log-socket\") pod \"ovnkube-node-b7b9v\" (UID: \"db22f88d-a953-488e-b384-67f991351f73\") " pod="openshift-ovn-kubernetes/ovnkube-node-b7b9v" Jul 09 15:31:20.637577 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:31:20.637271 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/db22f88d-a953-488e-b384-67f991351f73-host-run-ovn-kubernetes\") pod \"ovnkube-node-b7b9v\" (UID: \"db22f88d-a953-488e-b384-67f991351f73\") " pod="openshift-ovn-kubernetes/ovnkube-node-b7b9v" Jul 09 15:31:20.637577 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:31:20.637291 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/86ea7a2d-fc63-4660-a266-93fe5360fe49-host-var-lib-cni-bin\") pod \"multus-v27h9\" (UID: \"86ea7a2d-fc63-4660-a266-93fe5360fe49\") " pod="openshift-multus/multus-v27h9" Jul 09 15:31:20.637577 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:31:20.637307 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l6q9n\" (UniqueName: \"kubernetes.io/projected/e8de13cc-b0e0-4a54-bd44-3ba73cbf2bf2-kube-api-access-l6q9n\") pod \"network-check-target-sdbbl\" (UID: \"e8de13cc-b0e0-4a54-bd44-3ba73cbf2bf2\") " pod="openshift-network-diagnostics/network-check-target-sdbbl" Jul 09 15:31:20.637577 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:31:20.637327 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lgqqz\" (UniqueName: \"kubernetes.io/projected/07a170e7-eeff-4591-9a10-ecfe6a8e8100-kube-api-access-lgqqz\") pod \"iptables-alerter-7zxwk\" (UID: \"07a170e7-eeff-4591-9a10-ecfe6a8e8100\") " pod="openshift-network-operator/iptables-alerter-7zxwk" Jul 09 15:31:20.637577 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:31:20.637365 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/db22f88d-a953-488e-b384-67f991351f73-run-openvswitch\") pod \"ovnkube-node-b7b9v\" (UID: \"db22f88d-a953-488e-b384-67f991351f73\") " pod="openshift-ovn-kubernetes/ovnkube-node-b7b9v" Jul 09 15:31:20.638146 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:31:20.637396 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/db22f88d-a953-488e-b384-67f991351f73-host-cni-netd\") pod \"ovnkube-node-b7b9v\" (UID: \"db22f88d-a953-488e-b384-67f991351f73\") " pod="openshift-ovn-kubernetes/ovnkube-node-b7b9v" Jul 09 15:31:20.638146 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:31:20.637423 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/86ea7a2d-fc63-4660-a266-93fe5360fe49-multus-daemon-config\") pod \"multus-v27h9\" (UID: \"86ea7a2d-fc63-4660-a266-93fe5360fe49\") " pod="openshift-multus/multus-v27h9" Jul 09 15:31:20.638146 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:31:20.637465 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/86ea7a2d-fc63-4660-a266-93fe5360fe49-os-release\") pod \"multus-v27h9\" (UID: \"86ea7a2d-fc63-4660-a266-93fe5360fe49\") " pod="openshift-multus/multus-v27h9" Jul 09 15:31:20.638146 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:31:20.637487 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/85842275-e9c5-44af-8e99-1700a37602e3-cnibin\") pod \"multus-additional-cni-plugins-g5qvc\" (UID: \"85842275-e9c5-44af-8e99-1700a37602e3\") " pod="openshift-multus/multus-additional-cni-plugins-g5qvc" Jul 09 15:31:20.638146 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:31:20.637503 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/38e3cf1c-0139-47f3-a0f2-708c4e43b855-registration-dir\") pod \"aws-ebs-csi-driver-node-vxr6x\" (UID: \"38e3cf1c-0139-47f3-a0f2-708c4e43b855\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vxr6x" Jul 09 15:31:20.638146 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:31:20.637518 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/07a170e7-eeff-4591-9a10-ecfe6a8e8100-iptables-alerter-script\") pod \"iptables-alerter-7zxwk\" (UID: \"07a170e7-eeff-4591-9a10-ecfe6a8e8100\") " pod="openshift-network-operator/iptables-alerter-7zxwk" Jul 09 15:31:20.638146 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:31:20.637552 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/db22f88d-a953-488e-b384-67f991351f73-var-lib-openvswitch\") pod \"ovnkube-node-b7b9v\" (UID: \"db22f88d-a953-488e-b384-67f991351f73\") " pod="openshift-ovn-kubernetes/ovnkube-node-b7b9v" Jul 09 15:31:20.638146 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:31:20.637567 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/db22f88d-a953-488e-b384-67f991351f73-run-ovn\") pod \"ovnkube-node-b7b9v\" (UID: \"db22f88d-a953-488e-b384-67f991351f73\") " pod="openshift-ovn-kubernetes/ovnkube-node-b7b9v" Jul 09 15:31:20.638146 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:31:20.637585 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/86ea7a2d-fc63-4660-a266-93fe5360fe49-hostroot\") pod \"multus-v27h9\" (UID: \"86ea7a2d-fc63-4660-a266-93fe5360fe49\") " pod="openshift-multus/multus-v27h9" Jul 09 15:31:20.638146 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:31:20.637603 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/86ea7a2d-fc63-4660-a266-93fe5360fe49-multus-conf-dir\") pod \"multus-v27h9\" (UID: \"86ea7a2d-fc63-4660-a266-93fe5360fe49\") " pod="openshift-multus/multus-v27h9" Jul 09 15:31:20.638146 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:31:20.637643 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/38e3cf1c-0139-47f3-a0f2-708c4e43b855-kubelet-dir\") pod \"aws-ebs-csi-driver-node-vxr6x\" (UID: \"38e3cf1c-0139-47f3-a0f2-708c4e43b855\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vxr6x" Jul 09 15:31:20.640007 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:31:20.639988 2573 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Jul 09 15:31:20.663384 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:31:20.663364 2573 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-rnkl4" Jul 09 15:31:20.673495 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:31:20.673476 2573 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-rnkl4" Jul 09 15:31:20.725436 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:31:20.725400 2573 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world" Jul 09 15:31:20.738529 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:31:20.738494 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/db22f88d-a953-488e-b384-67f991351f73-host-run-ovn-kubernetes\") pod \"ovnkube-node-b7b9v\" (UID: \"db22f88d-a953-488e-b384-67f991351f73\") " pod="openshift-ovn-kubernetes/ovnkube-node-b7b9v" Jul 09 15:31:20.738529 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:31:20.738524 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/86ea7a2d-fc63-4660-a266-93fe5360fe49-host-var-lib-cni-bin\") pod \"multus-v27h9\" (UID: \"86ea7a2d-fc63-4660-a266-93fe5360fe49\") " pod="openshift-multus/multus-v27h9" Jul 09 15:31:20.738717 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:31:20.738546 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l6q9n\" (UniqueName: \"kubernetes.io/projected/e8de13cc-b0e0-4a54-bd44-3ba73cbf2bf2-kube-api-access-l6q9n\") pod \"network-check-target-sdbbl\" (UID: \"e8de13cc-b0e0-4a54-bd44-3ba73cbf2bf2\") " pod="openshift-network-diagnostics/network-check-target-sdbbl" Jul 09 15:31:20.738717 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:31:20.738562 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lgqqz\" (UniqueName: \"kubernetes.io/projected/07a170e7-eeff-4591-9a10-ecfe6a8e8100-kube-api-access-lgqqz\") pod \"iptables-alerter-7zxwk\" (UID: \"07a170e7-eeff-4591-9a10-ecfe6a8e8100\") " pod="openshift-network-operator/iptables-alerter-7zxwk" Jul 09 15:31:20.738717 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:31:20.738583 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/deaa6316-d948-414c-99b2-c4b07bbe68d8-run\") pod \"tuned-zp45w\" (UID: \"deaa6316-d948-414c-99b2-c4b07bbe68d8\") " pod="openshift-cluster-node-tuning-operator/tuned-zp45w" Jul 09 15:31:20.738717 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:31:20.738605 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/db22f88d-a953-488e-b384-67f991351f73-host-run-ovn-kubernetes\") pod \"ovnkube-node-b7b9v\" (UID: \"db22f88d-a953-488e-b384-67f991351f73\") " pod="openshift-ovn-kubernetes/ovnkube-node-b7b9v" Jul 09 15:31:20.738717 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:31:20.738683 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/deaa6316-d948-414c-99b2-c4b07bbe68d8-host\") pod \"tuned-zp45w\" (UID: \"deaa6316-d948-414c-99b2-c4b07bbe68d8\") " pod="openshift-cluster-node-tuning-operator/tuned-zp45w" Jul 09 15:31:20.738717 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:31:20.738679 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/86ea7a2d-fc63-4660-a266-93fe5360fe49-host-var-lib-cni-bin\") pod \"multus-v27h9\" (UID: \"86ea7a2d-fc63-4660-a266-93fe5360fe49\") " pod="openshift-multus/multus-v27h9" Jul 09 15:31:20.738717 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:31:20.738717 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/db22f88d-a953-488e-b384-67f991351f73-run-openvswitch\") pod \"ovnkube-node-b7b9v\" (UID: \"db22f88d-a953-488e-b384-67f991351f73\") " pod="openshift-ovn-kubernetes/ovnkube-node-b7b9v" Jul 09 15:31:20.739081 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:31:20.738743 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/db22f88d-a953-488e-b384-67f991351f73-host-cni-netd\") pod \"ovnkube-node-b7b9v\" (UID: \"db22f88d-a953-488e-b384-67f991351f73\") " pod="openshift-ovn-kubernetes/ovnkube-node-b7b9v" Jul 09 15:31:20.739081 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:31:20.738767 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/86ea7a2d-fc63-4660-a266-93fe5360fe49-multus-daemon-config\") pod \"multus-v27h9\" (UID: \"86ea7a2d-fc63-4660-a266-93fe5360fe49\") " pod="openshift-multus/multus-v27h9" Jul 09 15:31:20.739081 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:31:20.738787 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/deaa6316-d948-414c-99b2-c4b07bbe68d8-etc-sysconfig\") pod \"tuned-zp45w\" (UID: \"deaa6316-d948-414c-99b2-c4b07bbe68d8\") " pod="openshift-cluster-node-tuning-operator/tuned-zp45w" Jul 09 15:31:20.739081 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:31:20.738814 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/db22f88d-a953-488e-b384-67f991351f73-host-cni-netd\") pod \"ovnkube-node-b7b9v\" (UID: \"db22f88d-a953-488e-b384-67f991351f73\") " pod="openshift-ovn-kubernetes/ovnkube-node-b7b9v" Jul 09 15:31:20.739081 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:31:20.738851 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/86ea7a2d-fc63-4660-a266-93fe5360fe49-os-release\") pod \"multus-v27h9\" (UID: \"86ea7a2d-fc63-4660-a266-93fe5360fe49\") " pod="openshift-multus/multus-v27h9" Jul 09 15:31:20.739081 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:31:20.738857 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/db22f88d-a953-488e-b384-67f991351f73-run-openvswitch\") pod \"ovnkube-node-b7b9v\" (UID: \"db22f88d-a953-488e-b384-67f991351f73\") " pod="openshift-ovn-kubernetes/ovnkube-node-b7b9v" Jul 09 15:31:20.739081 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:31:20.738890 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/deaa6316-d948-414c-99b2-c4b07bbe68d8-tmp\") pod \"tuned-zp45w\" (UID: \"deaa6316-d948-414c-99b2-c4b07bbe68d8\") " pod="openshift-cluster-node-tuning-operator/tuned-zp45w" Jul 09 15:31:20.739081 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:31:20.738916 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/85842275-e9c5-44af-8e99-1700a37602e3-cnibin\") pod \"multus-additional-cni-plugins-g5qvc\" (UID: \"85842275-e9c5-44af-8e99-1700a37602e3\") " pod="openshift-multus/multus-additional-cni-plugins-g5qvc" Jul 09 15:31:20.739081 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:31:20.738957 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/86ea7a2d-fc63-4660-a266-93fe5360fe49-os-release\") pod \"multus-v27h9\" (UID: \"86ea7a2d-fc63-4660-a266-93fe5360fe49\") " pod="openshift-multus/multus-v27h9" Jul 09 15:31:20.739081 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:31:20.739023 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/38e3cf1c-0139-47f3-a0f2-708c4e43b855-registration-dir\") pod \"aws-ebs-csi-driver-node-vxr6x\" (UID: \"38e3cf1c-0139-47f3-a0f2-708c4e43b855\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vxr6x" Jul 09 15:31:20.739081 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:31:20.739043 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/07a170e7-eeff-4591-9a10-ecfe6a8e8100-iptables-alerter-script\") pod \"iptables-alerter-7zxwk\" (UID: \"07a170e7-eeff-4591-9a10-ecfe6a8e8100\") " pod="openshift-network-operator/iptables-alerter-7zxwk" Jul 09 15:31:20.739081 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:31:20.739062 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/85842275-e9c5-44af-8e99-1700a37602e3-cnibin\") pod \"multus-additional-cni-plugins-g5qvc\" (UID: \"85842275-e9c5-44af-8e99-1700a37602e3\") " pod="openshift-multus/multus-additional-cni-plugins-g5qvc" Jul 09 15:31:20.739506 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:31:20.739098 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/38e3cf1c-0139-47f3-a0f2-708c4e43b855-registration-dir\") pod \"aws-ebs-csi-driver-node-vxr6x\" (UID: \"38e3cf1c-0139-47f3-a0f2-708c4e43b855\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vxr6x" Jul 09 15:31:20.739506 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:31:20.739133 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/db22f88d-a953-488e-b384-67f991351f73-var-lib-openvswitch\") pod \"ovnkube-node-b7b9v\" (UID: \"db22f88d-a953-488e-b384-67f991351f73\") " pod="openshift-ovn-kubernetes/ovnkube-node-b7b9v" Jul 09 15:31:20.739506 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:31:20.739151 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/db22f88d-a953-488e-b384-67f991351f73-run-ovn\") pod \"ovnkube-node-b7b9v\" (UID: \"db22f88d-a953-488e-b384-67f991351f73\") " pod="openshift-ovn-kubernetes/ovnkube-node-b7b9v" Jul 09 15:31:20.739506 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:31:20.739185 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/86ea7a2d-fc63-4660-a266-93fe5360fe49-hostroot\") pod \"multus-v27h9\" (UID: \"86ea7a2d-fc63-4660-a266-93fe5360fe49\") " pod="openshift-multus/multus-v27h9" Jul 09 15:31:20.739506 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:31:20.739190 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/db22f88d-a953-488e-b384-67f991351f73-var-lib-openvswitch\") pod \"ovnkube-node-b7b9v\" (UID: \"db22f88d-a953-488e-b384-67f991351f73\") " pod="openshift-ovn-kubernetes/ovnkube-node-b7b9v" Jul 09 15:31:20.739506 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:31:20.739208 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/86ea7a2d-fc63-4660-a266-93fe5360fe49-multus-conf-dir\") pod \"multus-v27h9\" (UID: \"86ea7a2d-fc63-4660-a266-93fe5360fe49\") " pod="openshift-multus/multus-v27h9" Jul 09 15:31:20.739506 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:31:20.739252 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/db22f88d-a953-488e-b384-67f991351f73-run-ovn\") pod \"ovnkube-node-b7b9v\" (UID: \"db22f88d-a953-488e-b384-67f991351f73\") " pod="openshift-ovn-kubernetes/ovnkube-node-b7b9v" Jul 09 15:31:20.739506 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:31:20.739251 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/38e3cf1c-0139-47f3-a0f2-708c4e43b855-kubelet-dir\") pod \"aws-ebs-csi-driver-node-vxr6x\" (UID: \"38e3cf1c-0139-47f3-a0f2-708c4e43b855\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vxr6x" Jul 09 15:31:20.739506 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:31:20.739291 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/86ea7a2d-fc63-4660-a266-93fe5360fe49-hostroot\") pod \"multus-v27h9\" (UID: \"86ea7a2d-fc63-4660-a266-93fe5360fe49\") " pod="openshift-multus/multus-v27h9" Jul 09 15:31:20.739506 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:31:20.739325 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9ad54b5e-c82e-480c-814f-e1ab55bf32fb-tmp-dir\") pod \"node-resolver-b5c8h\" (UID: \"9ad54b5e-c82e-480c-814f-e1ab55bf32fb\") " pod="openshift-dns/node-resolver-b5c8h" Jul 09 15:31:20.739506 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:31:20.739330 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/86ea7a2d-fc63-4660-a266-93fe5360fe49-multus-conf-dir\") pod \"multus-v27h9\" (UID: \"86ea7a2d-fc63-4660-a266-93fe5360fe49\") " pod="openshift-multus/multus-v27h9" Jul 09 15:31:20.739506 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:31:20.739350 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/db22f88d-a953-488e-b384-67f991351f73-host-slash\") pod \"ovnkube-node-b7b9v\" (UID: \"db22f88d-a953-488e-b384-67f991351f73\") " pod="openshift-ovn-kubernetes/ovnkube-node-b7b9v" Jul 09 15:31:20.739506 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:31:20.739350 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/38e3cf1c-0139-47f3-a0f2-708c4e43b855-kubelet-dir\") pod \"aws-ebs-csi-driver-node-vxr6x\" (UID: \"38e3cf1c-0139-47f3-a0f2-708c4e43b855\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vxr6x" Jul 09 15:31:20.739863 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:31:20.739513 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/86ea7a2d-fc63-4660-a266-93fe5360fe49-multus-daemon-config\") pod \"multus-v27h9\" (UID: \"86ea7a2d-fc63-4660-a266-93fe5360fe49\") " pod="openshift-multus/multus-v27h9" Jul 09 15:31:20.739863 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:31:20.739728 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/07a170e7-eeff-4591-9a10-ecfe6a8e8100-iptables-alerter-script\") pod \"iptables-alerter-7zxwk\" (UID: \"07a170e7-eeff-4591-9a10-ecfe6a8e8100\") " pod="openshift-network-operator/iptables-alerter-7zxwk" Jul 09 15:31:20.739863 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:31:20.739847 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/db22f88d-a953-488e-b384-67f991351f73-host-slash\") pod \"ovnkube-node-b7b9v\" (UID: \"db22f88d-a953-488e-b384-67f991351f73\") " pod="openshift-ovn-kubernetes/ovnkube-node-b7b9v" Jul 09 15:31:20.739972 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:31:20.739890 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/db22f88d-a953-488e-b384-67f991351f73-ovnkube-script-lib\") pod \"ovnkube-node-b7b9v\" (UID: \"db22f88d-a953-488e-b384-67f991351f73\") " pod="openshift-ovn-kubernetes/ovnkube-node-b7b9v" Jul 09 15:31:20.739972 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:31:20.739919 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/86ea7a2d-fc63-4660-a266-93fe5360fe49-system-cni-dir\") pod \"multus-v27h9\" (UID: \"86ea7a2d-fc63-4660-a266-93fe5360fe49\") " pod="openshift-multus/multus-v27h9" Jul 09 15:31:20.739972 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:31:20.739944 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/86ea7a2d-fc63-4660-a266-93fe5360fe49-host-run-k8s-cni-cncf-io\") pod \"multus-v27h9\" (UID: \"86ea7a2d-fc63-4660-a266-93fe5360fe49\") " pod="openshift-multus/multus-v27h9" Jul 09 15:31:20.740115 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:31:20.739976 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9cw2h\" (UniqueName: \"kubernetes.io/projected/8e56f4e6-cbec-422f-9505-c7c6654159d9-kube-api-access-9cw2h\") pod \"network-metrics-daemon-8f2p4\" (UID: \"8e56f4e6-cbec-422f-9505-c7c6654159d9\") " pod="openshift-multus/network-metrics-daemon-8f2p4" Jul 09 15:31:20.740115 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:31:20.740006 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/deaa6316-d948-414c-99b2-c4b07bbe68d8-etc-systemd\") pod \"tuned-zp45w\" (UID: \"deaa6316-d948-414c-99b2-c4b07bbe68d8\") " pod="openshift-cluster-node-tuning-operator/tuned-zp45w" Jul 09 15:31:20.740115 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:31:20.740009 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/86ea7a2d-fc63-4660-a266-93fe5360fe49-system-cni-dir\") pod \"multus-v27h9\" (UID: \"86ea7a2d-fc63-4660-a266-93fe5360fe49\") " pod="openshift-multus/multus-v27h9" Jul 09 15:31:20.740115 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:31:20.740031 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/86ea7a2d-fc63-4660-a266-93fe5360fe49-host-run-k8s-cni-cncf-io\") pod \"multus-v27h9\" (UID: \"86ea7a2d-fc63-4660-a266-93fe5360fe49\") " pod="openshift-multus/multus-v27h9" Jul 09 15:31:20.740115 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:31:20.740035 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/db22f88d-a953-488e-b384-67f991351f73-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-b7b9v\" (UID: \"db22f88d-a953-488e-b384-67f991351f73\") " pod="openshift-ovn-kubernetes/ovnkube-node-b7b9v" Jul 09 15:31:20.740115 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:31:20.740074 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/db22f88d-a953-488e-b384-67f991351f73-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-b7b9v\" (UID: \"db22f88d-a953-488e-b384-67f991351f73\") " pod="openshift-ovn-kubernetes/ovnkube-node-b7b9v" Jul 09 15:31:20.740115 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:31:20.740078 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/86ea7a2d-fc63-4660-a266-93fe5360fe49-etc-kubernetes\") pod \"multus-v27h9\" (UID: \"86ea7a2d-fc63-4660-a266-93fe5360fe49\") " pod="openshift-multus/multus-v27h9" Jul 09 15:31:20.740115 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:31:20.740106 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/9ad54b5e-c82e-480c-814f-e1ab55bf32fb-hosts-file\") pod \"node-resolver-b5c8h\" (UID: \"9ad54b5e-c82e-480c-814f-e1ab55bf32fb\") " pod="openshift-dns/node-resolver-b5c8h" Jul 09 15:31:20.740115 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:31:20.740113 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/86ea7a2d-fc63-4660-a266-93fe5360fe49-etc-kubernetes\") pod \"multus-v27h9\" (UID: \"86ea7a2d-fc63-4660-a266-93fe5360fe49\") " pod="openshift-multus/multus-v27h9" Jul 09 15:31:20.740436 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:31:20.740136 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/deaa6316-d948-414c-99b2-c4b07bbe68d8-etc-sysctl-conf\") pod \"tuned-zp45w\" (UID: \"deaa6316-d948-414c-99b2-c4b07bbe68d8\") " pod="openshift-cluster-node-tuning-operator/tuned-zp45w" Jul 09 15:31:20.740436 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:31:20.740164 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-986n2\" (UniqueName: \"kubernetes.io/projected/deaa6316-d948-414c-99b2-c4b07bbe68d8-kube-api-access-986n2\") pod \"tuned-zp45w\" (UID: \"deaa6316-d948-414c-99b2-c4b07bbe68d8\") " pod="openshift-cluster-node-tuning-operator/tuned-zp45w" Jul 09 15:31:20.740436 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:31:20.740188 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/db22f88d-a953-488e-b384-67f991351f73-host-kubelet\") pod \"ovnkube-node-b7b9v\" (UID: \"db22f88d-a953-488e-b384-67f991351f73\") " pod="openshift-ovn-kubernetes/ovnkube-node-b7b9v" Jul 09 15:31:20.740436 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:31:20.740211 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/db22f88d-a953-488e-b384-67f991351f73-host-cni-bin\") pod \"ovnkube-node-b7b9v\" (UID: \"db22f88d-a953-488e-b384-67f991351f73\") " pod="openshift-ovn-kubernetes/ovnkube-node-b7b9v" Jul 09 15:31:20.740436 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:31:20.740247 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/85842275-e9c5-44af-8e99-1700a37602e3-system-cni-dir\") pod \"multus-additional-cni-plugins-g5qvc\" (UID: \"85842275-e9c5-44af-8e99-1700a37602e3\") " pod="openshift-multus/multus-additional-cni-plugins-g5qvc" Jul 09 15:31:20.740436 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:31:20.740271 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-df9gg\" (UniqueName: \"kubernetes.io/projected/86ea7a2d-fc63-4660-a266-93fe5360fe49-kube-api-access-df9gg\") pod \"multus-v27h9\" (UID: \"86ea7a2d-fc63-4660-a266-93fe5360fe49\") " pod="openshift-multus/multus-v27h9" Jul 09 15:31:20.740436 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:31:20.740288 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/db22f88d-a953-488e-b384-67f991351f73-host-kubelet\") pod \"ovnkube-node-b7b9v\" (UID: \"db22f88d-a953-488e-b384-67f991351f73\") " pod="openshift-ovn-kubernetes/ovnkube-node-b7b9v" Jul 09 15:31:20.740436 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:31:20.740300 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/85842275-e9c5-44af-8e99-1700a37602e3-system-cni-dir\") pod \"multus-additional-cni-plugins-g5qvc\" (UID: \"85842275-e9c5-44af-8e99-1700a37602e3\") " pod="openshift-multus/multus-additional-cni-plugins-g5qvc" Jul 09 15:31:20.740436 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:31:20.740304 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/38e3cf1c-0139-47f3-a0f2-708c4e43b855-device-dir\") pod \"aws-ebs-csi-driver-node-vxr6x\" (UID: \"38e3cf1c-0139-47f3-a0f2-708c4e43b855\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vxr6x" Jul 09 15:31:20.740436 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:31:20.740335 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/1094a821-3b3e-46d8-ac90-64c7ae6e8cad-serviceca\") pod \"node-ca-trnjd\" (UID: \"1094a821-3b3e-46d8-ac90-64c7ae6e8cad\") " pod="openshift-image-registry/node-ca-trnjd" Jul 09 15:31:20.740436 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:31:20.740347 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/38e3cf1c-0139-47f3-a0f2-708c4e43b855-device-dir\") pod \"aws-ebs-csi-driver-node-vxr6x\" (UID: \"38e3cf1c-0139-47f3-a0f2-708c4e43b855\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vxr6x" Jul 09 15:31:20.740436 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:31:20.740352 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8pzhh\" (UniqueName: \"kubernetes.io/projected/1094a821-3b3e-46d8-ac90-64c7ae6e8cad-kube-api-access-8pzhh\") pod \"node-ca-trnjd\" (UID: \"1094a821-3b3e-46d8-ac90-64c7ae6e8cad\") " pod="openshift-image-registry/node-ca-trnjd" Jul 09 15:31:20.740436 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:31:20.740302 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/db22f88d-a953-488e-b384-67f991351f73-host-cni-bin\") pod \"ovnkube-node-b7b9v\" (UID: \"db22f88d-a953-488e-b384-67f991351f73\") " pod="openshift-ovn-kubernetes/ovnkube-node-b7b9v" Jul 09 15:31:20.740436 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:31:20.740380 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/deaa6316-d948-414c-99b2-c4b07bbe68d8-etc-kubernetes\") pod \"tuned-zp45w\" (UID: \"deaa6316-d948-414c-99b2-c4b07bbe68d8\") " pod="openshift-cluster-node-tuning-operator/tuned-zp45w" Jul 09 15:31:20.740436 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:31:20.740420 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/85842275-e9c5-44af-8e99-1700a37602e3-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-g5qvc\" (UID: \"85842275-e9c5-44af-8e99-1700a37602e3\") " pod="openshift-multus/multus-additional-cni-plugins-g5qvc" Jul 09 15:31:20.741025 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:31:20.740463 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/86ea7a2d-fc63-4660-a266-93fe5360fe49-multus-cni-dir\") pod \"multus-v27h9\" (UID: \"86ea7a2d-fc63-4660-a266-93fe5360fe49\") " pod="openshift-multus/multus-v27h9" Jul 09 15:31:20.741025 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:31:20.740525 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/86ea7a2d-fc63-4660-a266-93fe5360fe49-host-var-lib-kubelet\") pod \"multus-v27h9\" (UID: \"86ea7a2d-fc63-4660-a266-93fe5360fe49\") " pod="openshift-multus/multus-v27h9" Jul 09 15:31:20.741025 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:31:20.740569 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/38e3cf1c-0139-47f3-a0f2-708c4e43b855-socket-dir\") pod \"aws-ebs-csi-driver-node-vxr6x\" (UID: \"38e3cf1c-0139-47f3-a0f2-708c4e43b855\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vxr6x" Jul 09 15:31:20.741025 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:31:20.740602 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/02be15c9-9800-4cae-9293-de04d5c845f8-agent-certs\") pod \"konnectivity-agent-zrrxr\" (UID: \"02be15c9-9800-4cae-9293-de04d5c845f8\") " pod="kube-system/konnectivity-agent-zrrxr" Jul 09 15:31:20.741025 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:31:20.740623 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s26wd\" (UniqueName: \"kubernetes.io/projected/9ad54b5e-c82e-480c-814f-e1ab55bf32fb-kube-api-access-s26wd\") pod \"node-resolver-b5c8h\" (UID: \"9ad54b5e-c82e-480c-814f-e1ab55bf32fb\") " pod="openshift-dns/node-resolver-b5c8h" Jul 09 15:31:20.741025 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:31:20.740636 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/86ea7a2d-fc63-4660-a266-93fe5360fe49-host-var-lib-kubelet\") pod \"multus-v27h9\" (UID: \"86ea7a2d-fc63-4660-a266-93fe5360fe49\") " pod="openshift-multus/multus-v27h9" Jul 09 15:31:20.741025 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:31:20.740650 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/86ea7a2d-fc63-4660-a266-93fe5360fe49-multus-cni-dir\") pod \"multus-v27h9\" (UID: \"86ea7a2d-fc63-4660-a266-93fe5360fe49\") " pod="openshift-multus/multus-v27h9" Jul 09 15:31:20.741025 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:31:20.740645 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/deaa6316-d948-414c-99b2-c4b07bbe68d8-lib-modules\") pod \"tuned-zp45w\" (UID: \"deaa6316-d948-414c-99b2-c4b07bbe68d8\") " pod="openshift-cluster-node-tuning-operator/tuned-zp45w" Jul 09 15:31:20.741025 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:31:20.740695 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/d9ea5934-356a-47e5-9fc6-1ae4f39f4e19-dbus\") pod \"global-pull-secret-syncer-h2cs6\" (UID: \"d9ea5934-356a-47e5-9fc6-1ae4f39f4e19\") " pod="kube-system/global-pull-secret-syncer-h2cs6" Jul 09 15:31:20.741025 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:31:20.740720 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/38e3cf1c-0139-47f3-a0f2-708c4e43b855-socket-dir\") pod \"aws-ebs-csi-driver-node-vxr6x\" (UID: \"38e3cf1c-0139-47f3-a0f2-708c4e43b855\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vxr6x" Jul 09 15:31:20.741025 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:31:20.740722 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/db22f88d-a953-488e-b384-67f991351f73-etc-openvswitch\") pod \"ovnkube-node-b7b9v\" (UID: \"db22f88d-a953-488e-b384-67f991351f73\") " pod="openshift-ovn-kubernetes/ovnkube-node-b7b9v" Jul 09 15:31:20.741025 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:31:20.740763 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/db22f88d-a953-488e-b384-67f991351f73-etc-openvswitch\") pod \"ovnkube-node-b7b9v\" (UID: \"db22f88d-a953-488e-b384-67f991351f73\") " pod="openshift-ovn-kubernetes/ovnkube-node-b7b9v" Jul 09 15:31:20.741025 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:31:20.740757 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/db22f88d-a953-488e-b384-67f991351f73-ovnkube-config\") pod \"ovnkube-node-b7b9v\" (UID: \"db22f88d-a953-488e-b384-67f991351f73\") " pod="openshift-ovn-kubernetes/ovnkube-node-b7b9v" Jul 09 15:31:20.741025 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:31:20.740807 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/86ea7a2d-fc63-4660-a266-93fe5360fe49-multus-socket-dir-parent\") pod \"multus-v27h9\" (UID: \"86ea7a2d-fc63-4660-a266-93fe5360fe49\") " pod="openshift-multus/multus-v27h9" Jul 09 15:31:20.741025 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:31:20.740833 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/86ea7a2d-fc63-4660-a266-93fe5360fe49-host-run-multus-certs\") pod \"multus-v27h9\" (UID: \"86ea7a2d-fc63-4660-a266-93fe5360fe49\") " pod="openshift-multus/multus-v27h9" Jul 09 15:31:20.741025 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:31:20.740886 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/86ea7a2d-fc63-4660-a266-93fe5360fe49-multus-socket-dir-parent\") pod \"multus-v27h9\" (UID: \"86ea7a2d-fc63-4660-a266-93fe5360fe49\") " pod="openshift-multus/multus-v27h9" Jul 09 15:31:20.741025 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:31:20.740890 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6lttq\" (UniqueName: \"kubernetes.io/projected/38e3cf1c-0139-47f3-a0f2-708c4e43b855-kube-api-access-6lttq\") pod \"aws-ebs-csi-driver-node-vxr6x\" (UID: \"38e3cf1c-0139-47f3-a0f2-708c4e43b855\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vxr6x" Jul 09 15:31:20.741761 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:31:20.740919 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/1094a821-3b3e-46d8-ac90-64c7ae6e8cad-host\") pod \"node-ca-trnjd\" (UID: \"1094a821-3b3e-46d8-ac90-64c7ae6e8cad\") " pod="openshift-image-registry/node-ca-trnjd" Jul 09 15:31:20.741761 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:31:20.740943 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/d9ea5934-356a-47e5-9fc6-1ae4f39f4e19-original-pull-secret\") pod \"global-pull-secret-syncer-h2cs6\" (UID: \"d9ea5934-356a-47e5-9fc6-1ae4f39f4e19\") " pod="kube-system/global-pull-secret-syncer-h2cs6" Jul 09 15:31:20.741761 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:31:20.740961 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/deaa6316-d948-414c-99b2-c4b07bbe68d8-etc-tuned\") pod \"tuned-zp45w\" (UID: \"deaa6316-d948-414c-99b2-c4b07bbe68d8\") " pod="openshift-cluster-node-tuning-operator/tuned-zp45w" Jul 09 15:31:20.741761 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:31:20.740958 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/86ea7a2d-fc63-4660-a266-93fe5360fe49-host-run-multus-certs\") pod \"multus-v27h9\" (UID: \"86ea7a2d-fc63-4660-a266-93fe5360fe49\") " pod="openshift-multus/multus-v27h9" Jul 09 15:31:20.741761 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:31:20.740972 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/85842275-e9c5-44af-8e99-1700a37602e3-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-g5qvc\" (UID: \"85842275-e9c5-44af-8e99-1700a37602e3\") " pod="openshift-multus/multus-additional-cni-plugins-g5qvc" Jul 09 15:31:20.741761 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:31:20.740982 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/db22f88d-a953-488e-b384-67f991351f73-env-overrides\") pod \"ovnkube-node-b7b9v\" (UID: \"db22f88d-a953-488e-b384-67f991351f73\") " pod="openshift-ovn-kubernetes/ovnkube-node-b7b9v" Jul 09 15:31:20.741761 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:31:20.741023 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/85842275-e9c5-44af-8e99-1700a37602e3-cni-binary-copy\") pod \"multus-additional-cni-plugins-g5qvc\" (UID: \"85842275-e9c5-44af-8e99-1700a37602e3\") " pod="openshift-multus/multus-additional-cni-plugins-g5qvc" Jul 09 15:31:20.741761 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:31:20.741039 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/85842275-e9c5-44af-8e99-1700a37602e3-tuning-conf-dir\") pod \"multus-additional-cni-plugins-g5qvc\" (UID: \"85842275-e9c5-44af-8e99-1700a37602e3\") " pod="openshift-multus/multus-additional-cni-plugins-g5qvc" Jul 09 15:31:20.741761 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:31:20.741053 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/deaa6316-d948-414c-99b2-c4b07bbe68d8-etc-sysctl-d\") pod \"tuned-zp45w\" (UID: \"deaa6316-d948-414c-99b2-c4b07bbe68d8\") " pod="openshift-cluster-node-tuning-operator/tuned-zp45w" Jul 09 15:31:20.741761 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:31:20.741080 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/86ea7a2d-fc63-4660-a266-93fe5360fe49-host-run-netns\") pod \"multus-v27h9\" (UID: \"86ea7a2d-fc63-4660-a266-93fe5360fe49\") " pod="openshift-multus/multus-v27h9" Jul 09 15:31:20.741761 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:31:20.741105 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/86ea7a2d-fc63-4660-a266-93fe5360fe49-host-var-lib-cni-multus\") pod \"multus-v27h9\" (UID: \"86ea7a2d-fc63-4660-a266-93fe5360fe49\") " pod="openshift-multus/multus-v27h9" Jul 09 15:31:20.741761 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:31:20.741131 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/38e3cf1c-0139-47f3-a0f2-708c4e43b855-sys-fs\") pod \"aws-ebs-csi-driver-node-vxr6x\" (UID: \"38e3cf1c-0139-47f3-a0f2-708c4e43b855\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vxr6x" Jul 09 15:31:20.741761 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:31:20.741154 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/07a170e7-eeff-4591-9a10-ecfe6a8e8100-host-slash\") pod \"iptables-alerter-7zxwk\" (UID: \"07a170e7-eeff-4591-9a10-ecfe6a8e8100\") " pod="openshift-network-operator/iptables-alerter-7zxwk" Jul 09 15:31:20.741761 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:31:20.741187 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/db22f88d-a953-488e-b384-67f991351f73-ovn-node-metrics-cert\") pod \"ovnkube-node-b7b9v\" (UID: \"db22f88d-a953-488e-b384-67f991351f73\") " pod="openshift-ovn-kubernetes/ovnkube-node-b7b9v" Jul 09 15:31:20.741761 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:31:20.741208 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-njbgg\" (UniqueName: \"kubernetes.io/projected/db22f88d-a953-488e-b384-67f991351f73-kube-api-access-njbgg\") pod \"ovnkube-node-b7b9v\" (UID: \"db22f88d-a953-488e-b384-67f991351f73\") " pod="openshift-ovn-kubernetes/ovnkube-node-b7b9v" Jul 09 15:31:20.741761 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:31:20.741217 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/86ea7a2d-fc63-4660-a266-93fe5360fe49-host-var-lib-cni-multus\") pod \"multus-v27h9\" (UID: \"86ea7a2d-fc63-4660-a266-93fe5360fe49\") " pod="openshift-multus/multus-v27h9" Jul 09 15:31:20.741761 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:31:20.741230 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/85842275-e9c5-44af-8e99-1700a37602e3-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-g5qvc\" (UID: \"85842275-e9c5-44af-8e99-1700a37602e3\") " pod="openshift-multus/multus-additional-cni-plugins-g5qvc" Jul 09 15:31:20.742418 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:31:20.741268 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/85842275-e9c5-44af-8e99-1700a37602e3-tuning-conf-dir\") pod \"multus-additional-cni-plugins-g5qvc\" (UID: \"85842275-e9c5-44af-8e99-1700a37602e3\") " pod="openshift-multus/multus-additional-cni-plugins-g5qvc" Jul 09 15:31:20.742418 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:31:20.741291 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ql4w6\" (UniqueName: \"kubernetes.io/projected/85842275-e9c5-44af-8e99-1700a37602e3-kube-api-access-ql4w6\") pod \"multus-additional-cni-plugins-g5qvc\" (UID: \"85842275-e9c5-44af-8e99-1700a37602e3\") " pod="openshift-multus/multus-additional-cni-plugins-g5qvc" Jul 09 15:31:20.742418 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:31:20.741309 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/db22f88d-a953-488e-b384-67f991351f73-ovnkube-config\") pod \"ovnkube-node-b7b9v\" (UID: \"db22f88d-a953-488e-b384-67f991351f73\") " pod="openshift-ovn-kubernetes/ovnkube-node-b7b9v" Jul 09 15:31:20.742418 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:31:20.741346 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/86ea7a2d-fc63-4660-a266-93fe5360fe49-cni-binary-copy\") pod \"multus-v27h9\" (UID: \"86ea7a2d-fc63-4660-a266-93fe5360fe49\") " pod="openshift-multus/multus-v27h9" Jul 09 15:31:20.742418 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:31:20.741389 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/db22f88d-a953-488e-b384-67f991351f73-ovnkube-script-lib\") pod \"ovnkube-node-b7b9v\" (UID: \"db22f88d-a953-488e-b384-67f991351f73\") " pod="openshift-ovn-kubernetes/ovnkube-node-b7b9v" Jul 09 15:31:20.742418 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:31:20.741407 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/db22f88d-a953-488e-b384-67f991351f73-env-overrides\") pod \"ovnkube-node-b7b9v\" (UID: \"db22f88d-a953-488e-b384-67f991351f73\") " pod="openshift-ovn-kubernetes/ovnkube-node-b7b9v" Jul 09 15:31:20.742418 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:31:20.741418 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/38e3cf1c-0139-47f3-a0f2-708c4e43b855-etc-selinux\") pod \"aws-ebs-csi-driver-node-vxr6x\" (UID: \"38e3cf1c-0139-47f3-a0f2-708c4e43b855\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vxr6x" Jul 09 15:31:20.742418 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:31:20.741434 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/86ea7a2d-fc63-4660-a266-93fe5360fe49-host-run-netns\") pod \"multus-v27h9\" (UID: \"86ea7a2d-fc63-4660-a266-93fe5360fe49\") " pod="openshift-multus/multus-v27h9" Jul 09 15:31:20.742418 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:31:20.741448 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8e56f4e6-cbec-422f-9505-c7c6654159d9-metrics-certs\") pod \"network-metrics-daemon-8f2p4\" (UID: \"8e56f4e6-cbec-422f-9505-c7c6654159d9\") " pod="openshift-multus/network-metrics-daemon-8f2p4" Jul 09 15:31:20.742418 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:31:20.741472 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/deaa6316-d948-414c-99b2-c4b07bbe68d8-sys\") pod \"tuned-zp45w\" (UID: \"deaa6316-d948-414c-99b2-c4b07bbe68d8\") " pod="openshift-cluster-node-tuning-operator/tuned-zp45w" Jul 09 15:31:20.742418 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:31:20.741519 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/db22f88d-a953-488e-b384-67f991351f73-systemd-units\") pod \"ovnkube-node-b7b9v\" (UID: \"db22f88d-a953-488e-b384-67f991351f73\") " pod="openshift-ovn-kubernetes/ovnkube-node-b7b9v" Jul 09 15:31:20.742418 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:31:20.741559 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/38e3cf1c-0139-47f3-a0f2-708c4e43b855-sys-fs\") pod \"aws-ebs-csi-driver-node-vxr6x\" (UID: \"38e3cf1c-0139-47f3-a0f2-708c4e43b855\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vxr6x" Jul 09 15:31:20.742418 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:31:20.741562 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/85842275-e9c5-44af-8e99-1700a37602e3-cni-binary-copy\") pod \"multus-additional-cni-plugins-g5qvc\" (UID: \"85842275-e9c5-44af-8e99-1700a37602e3\") " pod="openshift-multus/multus-additional-cni-plugins-g5qvc" Jul 09 15:31:20.742418 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:31:20.741571 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/db22f88d-a953-488e-b384-67f991351f73-run-systemd\") pod \"ovnkube-node-b7b9v\" (UID: \"db22f88d-a953-488e-b384-67f991351f73\") " pod="openshift-ovn-kubernetes/ovnkube-node-b7b9v" Jul 09 15:31:20.742418 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:31:20.741615 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/07a170e7-eeff-4591-9a10-ecfe6a8e8100-host-slash\") pod \"iptables-alerter-7zxwk\" (UID: \"07a170e7-eeff-4591-9a10-ecfe6a8e8100\") " pod="openshift-network-operator/iptables-alerter-7zxwk" Jul 09 15:31:20.742418 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:31:20.741617 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/db22f88d-a953-488e-b384-67f991351f73-systemd-units\") pod \"ovnkube-node-b7b9v\" (UID: \"db22f88d-a953-488e-b384-67f991351f73\") " pod="openshift-ovn-kubernetes/ovnkube-node-b7b9v" Jul 09 15:31:20.742418 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:31:20.741632 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/db22f88d-a953-488e-b384-67f991351f73-node-log\") pod \"ovnkube-node-b7b9v\" (UID: \"db22f88d-a953-488e-b384-67f991351f73\") " pod="openshift-ovn-kubernetes/ovnkube-node-b7b9v" Jul 09 15:31:20.742418 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:31:20.741617 2573 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory" Jul 09 15:31:20.743224 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:31:20.741677 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/db22f88d-a953-488e-b384-67f991351f73-run-systemd\") pod \"ovnkube-node-b7b9v\" (UID: \"db22f88d-a953-488e-b384-67f991351f73\") " pod="openshift-ovn-kubernetes/ovnkube-node-b7b9v" Jul 09 15:31:20.743224 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:31:20.741680 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/db22f88d-a953-488e-b384-67f991351f73-node-log\") pod \"ovnkube-node-b7b9v\" (UID: \"db22f88d-a953-488e-b384-67f991351f73\") " pod="openshift-ovn-kubernetes/ovnkube-node-b7b9v" Jul 09 15:31:20.743224 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:31:20.741720 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/85842275-e9c5-44af-8e99-1700a37602e3-os-release\") pod \"multus-additional-cni-plugins-g5qvc\" (UID: \"85842275-e9c5-44af-8e99-1700a37602e3\") " pod="openshift-multus/multus-additional-cni-plugins-g5qvc" Jul 09 15:31:20.743224 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:31:20.741733 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/38e3cf1c-0139-47f3-a0f2-708c4e43b855-etc-selinux\") pod \"aws-ebs-csi-driver-node-vxr6x\" (UID: \"38e3cf1c-0139-47f3-a0f2-708c4e43b855\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vxr6x" Jul 09 15:31:20.743224 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:31:20.741756 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/85842275-e9c5-44af-8e99-1700a37602e3-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-g5qvc\" (UID: \"85842275-e9c5-44af-8e99-1700a37602e3\") " pod="openshift-multus/multus-additional-cni-plugins-g5qvc" Jul 09 15:31:20.743224 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:31:20.741760 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/d9ea5934-356a-47e5-9fc6-1ae4f39f4e19-kubelet-config\") pod \"global-pull-secret-syncer-h2cs6\" (UID: \"d9ea5934-356a-47e5-9fc6-1ae4f39f4e19\") " pod="kube-system/global-pull-secret-syncer-h2cs6" Jul 09 15:31:20.743224 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:31:20.741797 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/85842275-e9c5-44af-8e99-1700a37602e3-os-release\") pod \"multus-additional-cni-plugins-g5qvc\" (UID: \"85842275-e9c5-44af-8e99-1700a37602e3\") " pod="openshift-multus/multus-additional-cni-plugins-g5qvc" Jul 09 15:31:20.743224 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:31:20.741827 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/db22f88d-a953-488e-b384-67f991351f73-host-run-netns\") pod \"ovnkube-node-b7b9v\" (UID: \"db22f88d-a953-488e-b384-67f991351f73\") " pod="openshift-ovn-kubernetes/ovnkube-node-b7b9v" Jul 09 15:31:20.743224 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:31:20.741844 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/86ea7a2d-fc63-4660-a266-93fe5360fe49-cnibin\") pod \"multus-v27h9\" (UID: \"86ea7a2d-fc63-4660-a266-93fe5360fe49\") " pod="openshift-multus/multus-v27h9" Jul 09 15:31:20.743224 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:31:20.741860 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/02be15c9-9800-4cae-9293-de04d5c845f8-konnectivity-ca\") pod \"konnectivity-agent-zrrxr\" (UID: \"02be15c9-9800-4cae-9293-de04d5c845f8\") " pod="kube-system/konnectivity-agent-zrrxr" Jul 09 15:31:20.743224 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:31:20.741862 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/db22f88d-a953-488e-b384-67f991351f73-host-run-netns\") pod \"ovnkube-node-b7b9v\" (UID: \"db22f88d-a953-488e-b384-67f991351f73\") " pod="openshift-ovn-kubernetes/ovnkube-node-b7b9v" Jul 09 15:31:20.743224 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:31:20.742269 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/86ea7a2d-fc63-4660-a266-93fe5360fe49-cnibin\") pod \"multus-v27h9\" (UID: \"86ea7a2d-fc63-4660-a266-93fe5360fe49\") " pod="openshift-multus/multus-v27h9" Jul 09 15:31:20.743224 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:31:20.742398 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/deaa6316-d948-414c-99b2-c4b07bbe68d8-etc-modprobe-d\") pod \"tuned-zp45w\" (UID: \"deaa6316-d948-414c-99b2-c4b07bbe68d8\") " pod="openshift-cluster-node-tuning-operator/tuned-zp45w" Jul 09 15:31:20.743224 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:31:20.742436 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/deaa6316-d948-414c-99b2-c4b07bbe68d8-var-lib-kubelet\") pod \"tuned-zp45w\" (UID: \"deaa6316-d948-414c-99b2-c4b07bbe68d8\") " pod="openshift-cluster-node-tuning-operator/tuned-zp45w" Jul 09 15:31:20.743224 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:31:20.742468 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/db22f88d-a953-488e-b384-67f991351f73-log-socket\") pod \"ovnkube-node-b7b9v\" (UID: \"db22f88d-a953-488e-b384-67f991351f73\") " pod="openshift-ovn-kubernetes/ovnkube-node-b7b9v" Jul 09 15:31:20.743224 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:31:20.742607 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/db22f88d-a953-488e-b384-67f991351f73-log-socket\") pod \"ovnkube-node-b7b9v\" (UID: \"db22f88d-a953-488e-b384-67f991351f73\") " pod="openshift-ovn-kubernetes/ovnkube-node-b7b9v" Jul 09 15:31:20.743224 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:31:20.743194 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/86ea7a2d-fc63-4660-a266-93fe5360fe49-cni-binary-copy\") pod \"multus-v27h9\" (UID: \"86ea7a2d-fc63-4660-a266-93fe5360fe49\") " pod="openshift-multus/multus-v27h9" Jul 09 15:31:20.745757 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:31:20.745425 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/db22f88d-a953-488e-b384-67f991351f73-ovn-node-metrics-cert\") pod \"ovnkube-node-b7b9v\" (UID: \"db22f88d-a953-488e-b384-67f991351f73\") " pod="openshift-ovn-kubernetes/ovnkube-node-b7b9v" Jul 09 15:31:20.758557 ip-10-0-133-183 kubenswrapper[2573]: E0709 15:31:20.758530 2573 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 09 15:31:20.758557 ip-10-0-133-183 kubenswrapper[2573]: E0709 15:31:20.758556 2573 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 09 15:31:20.758747 ip-10-0-133-183 kubenswrapper[2573]: E0709 15:31:20.758569 2573 projected.go:196] Error preparing data for projected volume kube-api-access-l6q9n for pod openshift-network-diagnostics/network-check-target-sdbbl: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 09 15:31:20.758747 ip-10-0-133-183 kubenswrapper[2573]: E0709 15:31:20.758628 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/e8de13cc-b0e0-4a54-bd44-3ba73cbf2bf2-kube-api-access-l6q9n podName:e8de13cc-b0e0-4a54-bd44-3ba73cbf2bf2 nodeName:}" failed. No retries permitted until 2026-07-09 15:31:21.25860904 +0000 UTC m=+2.980115892 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-l6q9n" (UniqueName: "kubernetes.io/projected/e8de13cc-b0e0-4a54-bd44-3ba73cbf2bf2-kube-api-access-l6q9n") pod "network-check-target-sdbbl" (UID: "e8de13cc-b0e0-4a54-bd44-3ba73cbf2bf2") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 09 15:31:20.761293 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:31:20.761257 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-df9gg\" (UniqueName: \"kubernetes.io/projected/86ea7a2d-fc63-4660-a266-93fe5360fe49-kube-api-access-df9gg\") pod \"multus-v27h9\" (UID: \"86ea7a2d-fc63-4660-a266-93fe5360fe49\") " pod="openshift-multus/multus-v27h9" Jul 09 15:31:20.761419 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:31:20.761260 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ql4w6\" (UniqueName: \"kubernetes.io/projected/85842275-e9c5-44af-8e99-1700a37602e3-kube-api-access-ql4w6\") pod \"multus-additional-cni-plugins-g5qvc\" (UID: \"85842275-e9c5-44af-8e99-1700a37602e3\") " pod="openshift-multus/multus-additional-cni-plugins-g5qvc" Jul 09 15:31:20.761475 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:31:20.761426 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lgqqz\" (UniqueName: \"kubernetes.io/projected/07a170e7-eeff-4591-9a10-ecfe6a8e8100-kube-api-access-lgqqz\") pod \"iptables-alerter-7zxwk\" (UID: \"07a170e7-eeff-4591-9a10-ecfe6a8e8100\") " pod="openshift-network-operator/iptables-alerter-7zxwk" Jul 09 15:31:20.761627 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:31:20.761605 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-njbgg\" (UniqueName: \"kubernetes.io/projected/db22f88d-a953-488e-b384-67f991351f73-kube-api-access-njbgg\") pod \"ovnkube-node-b7b9v\" (UID: \"db22f88d-a953-488e-b384-67f991351f73\") " pod="openshift-ovn-kubernetes/ovnkube-node-b7b9v" Jul 09 15:31:20.761690 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:31:20.761648 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6lttq\" (UniqueName: \"kubernetes.io/projected/38e3cf1c-0139-47f3-a0f2-708c4e43b855-kube-api-access-6lttq\") pod \"aws-ebs-csi-driver-node-vxr6x\" (UID: \"38e3cf1c-0139-47f3-a0f2-708c4e43b855\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vxr6x" Jul 09 15:31:20.810860 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:31:20.810810 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-133-183.ec2.internal" event={"ID":"06227558853f1ac07df9b5c4930c43af","Type":"ContainerStarted","Data":"324e0c6e153c67ec232015060c7a16578a083e3d3f6c3abba6ab7675b562b0e9"} Jul 09 15:31:20.843806 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:31:20.843780 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9ad54b5e-c82e-480c-814f-e1ab55bf32fb-tmp-dir\") pod \"node-resolver-b5c8h\" (UID: \"9ad54b5e-c82e-480c-814f-e1ab55bf32fb\") " pod="openshift-dns/node-resolver-b5c8h" Jul 09 15:31:20.843917 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:31:20.843824 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9cw2h\" (UniqueName: \"kubernetes.io/projected/8e56f4e6-cbec-422f-9505-c7c6654159d9-kube-api-access-9cw2h\") pod \"network-metrics-daemon-8f2p4\" (UID: \"8e56f4e6-cbec-422f-9505-c7c6654159d9\") " pod="openshift-multus/network-metrics-daemon-8f2p4" Jul 09 15:31:20.843917 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:31:20.843847 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/deaa6316-d948-414c-99b2-c4b07bbe68d8-etc-systemd\") pod \"tuned-zp45w\" (UID: \"deaa6316-d948-414c-99b2-c4b07bbe68d8\") " pod="openshift-cluster-node-tuning-operator/tuned-zp45w" Jul 09 15:31:20.843917 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:31:20.843865 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/9ad54b5e-c82e-480c-814f-e1ab55bf32fb-hosts-file\") pod \"node-resolver-b5c8h\" (UID: \"9ad54b5e-c82e-480c-814f-e1ab55bf32fb\") " pod="openshift-dns/node-resolver-b5c8h" Jul 09 15:31:20.843917 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:31:20.843896 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/deaa6316-d948-414c-99b2-c4b07bbe68d8-etc-sysctl-conf\") pod \"tuned-zp45w\" (UID: \"deaa6316-d948-414c-99b2-c4b07bbe68d8\") " pod="openshift-cluster-node-tuning-operator/tuned-zp45w" Jul 09 15:31:20.844102 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:31:20.843919 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-986n2\" (UniqueName: \"kubernetes.io/projected/deaa6316-d948-414c-99b2-c4b07bbe68d8-kube-api-access-986n2\") pod \"tuned-zp45w\" (UID: \"deaa6316-d948-414c-99b2-c4b07bbe68d8\") " pod="openshift-cluster-node-tuning-operator/tuned-zp45w" Jul 09 15:31:20.844102 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:31:20.843971 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/9ad54b5e-c82e-480c-814f-e1ab55bf32fb-hosts-file\") pod \"node-resolver-b5c8h\" (UID: \"9ad54b5e-c82e-480c-814f-e1ab55bf32fb\") " pod="openshift-dns/node-resolver-b5c8h" Jul 09 15:31:20.844102 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:31:20.843971 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/deaa6316-d948-414c-99b2-c4b07bbe68d8-etc-systemd\") pod \"tuned-zp45w\" (UID: \"deaa6316-d948-414c-99b2-c4b07bbe68d8\") " pod="openshift-cluster-node-tuning-operator/tuned-zp45w" Jul 09 15:31:20.844102 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:31:20.843999 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/1094a821-3b3e-46d8-ac90-64c7ae6e8cad-serviceca\") pod \"node-ca-trnjd\" (UID: \"1094a821-3b3e-46d8-ac90-64c7ae6e8cad\") " pod="openshift-image-registry/node-ca-trnjd" Jul 09 15:31:20.844102 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:31:20.844040 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8pzhh\" (UniqueName: \"kubernetes.io/projected/1094a821-3b3e-46d8-ac90-64c7ae6e8cad-kube-api-access-8pzhh\") pod \"node-ca-trnjd\" (UID: \"1094a821-3b3e-46d8-ac90-64c7ae6e8cad\") " pod="openshift-image-registry/node-ca-trnjd" Jul 09 15:31:20.844102 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:31:20.844065 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/deaa6316-d948-414c-99b2-c4b07bbe68d8-etc-kubernetes\") pod \"tuned-zp45w\" (UID: \"deaa6316-d948-414c-99b2-c4b07bbe68d8\") " pod="openshift-cluster-node-tuning-operator/tuned-zp45w" Jul 09 15:31:20.844102 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:31:20.844078 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/deaa6316-d948-414c-99b2-c4b07bbe68d8-etc-sysctl-conf\") pod \"tuned-zp45w\" (UID: \"deaa6316-d948-414c-99b2-c4b07bbe68d8\") " pod="openshift-cluster-node-tuning-operator/tuned-zp45w" Jul 09 15:31:20.844102 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:31:20.844101 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/02be15c9-9800-4cae-9293-de04d5c845f8-agent-certs\") pod \"konnectivity-agent-zrrxr\" (UID: \"02be15c9-9800-4cae-9293-de04d5c845f8\") " pod="kube-system/konnectivity-agent-zrrxr" Jul 09 15:31:20.844463 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:31:20.844140 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s26wd\" (UniqueName: \"kubernetes.io/projected/9ad54b5e-c82e-480c-814f-e1ab55bf32fb-kube-api-access-s26wd\") pod \"node-resolver-b5c8h\" (UID: \"9ad54b5e-c82e-480c-814f-e1ab55bf32fb\") " pod="openshift-dns/node-resolver-b5c8h" Jul 09 15:31:20.844463 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:31:20.844150 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/deaa6316-d948-414c-99b2-c4b07bbe68d8-etc-kubernetes\") pod \"tuned-zp45w\" (UID: \"deaa6316-d948-414c-99b2-c4b07bbe68d8\") " pod="openshift-cluster-node-tuning-operator/tuned-zp45w" Jul 09 15:31:20.844463 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:31:20.844176 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/deaa6316-d948-414c-99b2-c4b07bbe68d8-lib-modules\") pod \"tuned-zp45w\" (UID: \"deaa6316-d948-414c-99b2-c4b07bbe68d8\") " pod="openshift-cluster-node-tuning-operator/tuned-zp45w" Jul 09 15:31:20.844463 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:31:20.844195 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9ad54b5e-c82e-480c-814f-e1ab55bf32fb-tmp-dir\") pod \"node-resolver-b5c8h\" (UID: \"9ad54b5e-c82e-480c-814f-e1ab55bf32fb\") " pod="openshift-dns/node-resolver-b5c8h" Jul 09 15:31:20.844463 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:31:20.844203 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/d9ea5934-356a-47e5-9fc6-1ae4f39f4e19-dbus\") pod \"global-pull-secret-syncer-h2cs6\" (UID: \"d9ea5934-356a-47e5-9fc6-1ae4f39f4e19\") " pod="kube-system/global-pull-secret-syncer-h2cs6" Jul 09 15:31:20.844463 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:31:20.844232 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/1094a821-3b3e-46d8-ac90-64c7ae6e8cad-host\") pod \"node-ca-trnjd\" (UID: \"1094a821-3b3e-46d8-ac90-64c7ae6e8cad\") " pod="openshift-image-registry/node-ca-trnjd" Jul 09 15:31:20.844463 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:31:20.844254 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/d9ea5934-356a-47e5-9fc6-1ae4f39f4e19-original-pull-secret\") pod \"global-pull-secret-syncer-h2cs6\" (UID: \"d9ea5934-356a-47e5-9fc6-1ae4f39f4e19\") " pod="kube-system/global-pull-secret-syncer-h2cs6" Jul 09 15:31:20.844463 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:31:20.844279 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/deaa6316-d948-414c-99b2-c4b07bbe68d8-etc-tuned\") pod \"tuned-zp45w\" (UID: \"deaa6316-d948-414c-99b2-c4b07bbe68d8\") " pod="openshift-cluster-node-tuning-operator/tuned-zp45w" Jul 09 15:31:20.844463 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:31:20.844301 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/deaa6316-d948-414c-99b2-c4b07bbe68d8-etc-sysctl-d\") pod \"tuned-zp45w\" (UID: \"deaa6316-d948-414c-99b2-c4b07bbe68d8\") " pod="openshift-cluster-node-tuning-operator/tuned-zp45w" Jul 09 15:31:20.844463 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:31:20.844302 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/1094a821-3b3e-46d8-ac90-64c7ae6e8cad-host\") pod \"node-ca-trnjd\" (UID: \"1094a821-3b3e-46d8-ac90-64c7ae6e8cad\") " pod="openshift-image-registry/node-ca-trnjd" Jul 09 15:31:20.844463 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:31:20.844343 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/d9ea5934-356a-47e5-9fc6-1ae4f39f4e19-dbus\") pod \"global-pull-secret-syncer-h2cs6\" (UID: \"d9ea5934-356a-47e5-9fc6-1ae4f39f4e19\") " pod="kube-system/global-pull-secret-syncer-h2cs6" Jul 09 15:31:20.844463 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:31:20.844354 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/deaa6316-d948-414c-99b2-c4b07bbe68d8-lib-modules\") pod \"tuned-zp45w\" (UID: \"deaa6316-d948-414c-99b2-c4b07bbe68d8\") " pod="openshift-cluster-node-tuning-operator/tuned-zp45w" Jul 09 15:31:20.844463 ip-10-0-133-183 kubenswrapper[2573]: E0709 15:31:20.844393 2573 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 09 15:31:20.844463 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:31:20.844437 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8e56f4e6-cbec-422f-9505-c7c6654159d9-metrics-certs\") pod \"network-metrics-daemon-8f2p4\" (UID: \"8e56f4e6-cbec-422f-9505-c7c6654159d9\") " pod="openshift-multus/network-metrics-daemon-8f2p4" Jul 09 15:31:20.844463 ip-10-0-133-183 kubenswrapper[2573]: E0709 15:31:20.844458 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d9ea5934-356a-47e5-9fc6-1ae4f39f4e19-original-pull-secret podName:d9ea5934-356a-47e5-9fc6-1ae4f39f4e19 nodeName:}" failed. No retries permitted until 2026-07-09 15:31:21.344440665 +0000 UTC m=+3.065947529 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/d9ea5934-356a-47e5-9fc6-1ae4f39f4e19-original-pull-secret") pod "global-pull-secret-syncer-h2cs6" (UID: "d9ea5934-356a-47e5-9fc6-1ae4f39f4e19") : object "kube-system"/"original-pull-secret" not registered Jul 09 15:31:20.845060 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:31:20.844479 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/deaa6316-d948-414c-99b2-c4b07bbe68d8-sys\") pod \"tuned-zp45w\" (UID: \"deaa6316-d948-414c-99b2-c4b07bbe68d8\") " pod="openshift-cluster-node-tuning-operator/tuned-zp45w" Jul 09 15:31:20.845060 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:31:20.844489 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/1094a821-3b3e-46d8-ac90-64c7ae6e8cad-serviceca\") pod \"node-ca-trnjd\" (UID: \"1094a821-3b3e-46d8-ac90-64c7ae6e8cad\") " pod="openshift-image-registry/node-ca-trnjd" Jul 09 15:31:20.845060 ip-10-0-133-183 kubenswrapper[2573]: E0709 15:31:20.844512 2573 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 09 15:31:20.845060 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:31:20.844522 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/d9ea5934-356a-47e5-9fc6-1ae4f39f4e19-kubelet-config\") pod \"global-pull-secret-syncer-h2cs6\" (UID: \"d9ea5934-356a-47e5-9fc6-1ae4f39f4e19\") " pod="kube-system/global-pull-secret-syncer-h2cs6" Jul 09 15:31:20.845060 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:31:20.844548 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/02be15c9-9800-4cae-9293-de04d5c845f8-konnectivity-ca\") pod \"konnectivity-agent-zrrxr\" (UID: \"02be15c9-9800-4cae-9293-de04d5c845f8\") " pod="kube-system/konnectivity-agent-zrrxr" Jul 09 15:31:20.845060 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:31:20.844559 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/deaa6316-d948-414c-99b2-c4b07bbe68d8-sys\") pod \"tuned-zp45w\" (UID: \"deaa6316-d948-414c-99b2-c4b07bbe68d8\") " pod="openshift-cluster-node-tuning-operator/tuned-zp45w" Jul 09 15:31:20.845060 ip-10-0-133-183 kubenswrapper[2573]: E0709 15:31:20.844573 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8e56f4e6-cbec-422f-9505-c7c6654159d9-metrics-certs podName:8e56f4e6-cbec-422f-9505-c7c6654159d9 nodeName:}" failed. No retries permitted until 2026-07-09 15:31:21.344556783 +0000 UTC m=+3.066063645 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/8e56f4e6-cbec-422f-9505-c7c6654159d9-metrics-certs") pod "network-metrics-daemon-8f2p4" (UID: "8e56f4e6-cbec-422f-9505-c7c6654159d9") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 09 15:31:20.845060 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:31:20.844561 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/d9ea5934-356a-47e5-9fc6-1ae4f39f4e19-kubelet-config\") pod \"global-pull-secret-syncer-h2cs6\" (UID: \"d9ea5934-356a-47e5-9fc6-1ae4f39f4e19\") " pod="kube-system/global-pull-secret-syncer-h2cs6" Jul 09 15:31:20.845060 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:31:20.844603 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/deaa6316-d948-414c-99b2-c4b07bbe68d8-etc-sysctl-d\") pod \"tuned-zp45w\" (UID: \"deaa6316-d948-414c-99b2-c4b07bbe68d8\") " pod="openshift-cluster-node-tuning-operator/tuned-zp45w" Jul 09 15:31:20.845060 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:31:20.844611 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/deaa6316-d948-414c-99b2-c4b07bbe68d8-etc-modprobe-d\") pod \"tuned-zp45w\" (UID: \"deaa6316-d948-414c-99b2-c4b07bbe68d8\") " pod="openshift-cluster-node-tuning-operator/tuned-zp45w" Jul 09 15:31:20.845060 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:31:20.844636 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/deaa6316-d948-414c-99b2-c4b07bbe68d8-var-lib-kubelet\") pod \"tuned-zp45w\" (UID: \"deaa6316-d948-414c-99b2-c4b07bbe68d8\") " pod="openshift-cluster-node-tuning-operator/tuned-zp45w" Jul 09 15:31:20.845060 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:31:20.844676 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/deaa6316-d948-414c-99b2-c4b07bbe68d8-run\") pod \"tuned-zp45w\" (UID: \"deaa6316-d948-414c-99b2-c4b07bbe68d8\") " pod="openshift-cluster-node-tuning-operator/tuned-zp45w" Jul 09 15:31:20.845060 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:31:20.844687 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/deaa6316-d948-414c-99b2-c4b07bbe68d8-etc-modprobe-d\") pod \"tuned-zp45w\" (UID: \"deaa6316-d948-414c-99b2-c4b07bbe68d8\") " pod="openshift-cluster-node-tuning-operator/tuned-zp45w" Jul 09 15:31:20.845060 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:31:20.844696 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/deaa6316-d948-414c-99b2-c4b07bbe68d8-host\") pod \"tuned-zp45w\" (UID: \"deaa6316-d948-414c-99b2-c4b07bbe68d8\") " pod="openshift-cluster-node-tuning-operator/tuned-zp45w" Jul 09 15:31:20.845060 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:31:20.844701 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/deaa6316-d948-414c-99b2-c4b07bbe68d8-var-lib-kubelet\") pod \"tuned-zp45w\" (UID: \"deaa6316-d948-414c-99b2-c4b07bbe68d8\") " pod="openshift-cluster-node-tuning-operator/tuned-zp45w" Jul 09 15:31:20.845060 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:31:20.844720 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/deaa6316-d948-414c-99b2-c4b07bbe68d8-etc-sysconfig\") pod \"tuned-zp45w\" (UID: \"deaa6316-d948-414c-99b2-c4b07bbe68d8\") " pod="openshift-cluster-node-tuning-operator/tuned-zp45w" Jul 09 15:31:20.845060 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:31:20.844738 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/deaa6316-d948-414c-99b2-c4b07bbe68d8-run\") pod \"tuned-zp45w\" (UID: \"deaa6316-d948-414c-99b2-c4b07bbe68d8\") " pod="openshift-cluster-node-tuning-operator/tuned-zp45w" Jul 09 15:31:20.845060 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:31:20.844746 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/deaa6316-d948-414c-99b2-c4b07bbe68d8-tmp\") pod \"tuned-zp45w\" (UID: \"deaa6316-d948-414c-99b2-c4b07bbe68d8\") " pod="openshift-cluster-node-tuning-operator/tuned-zp45w" Jul 09 15:31:20.845962 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:31:20.844758 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/deaa6316-d948-414c-99b2-c4b07bbe68d8-host\") pod \"tuned-zp45w\" (UID: \"deaa6316-d948-414c-99b2-c4b07bbe68d8\") " pod="openshift-cluster-node-tuning-operator/tuned-zp45w" Jul 09 15:31:20.845962 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:31:20.844782 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/deaa6316-d948-414c-99b2-c4b07bbe68d8-etc-sysconfig\") pod \"tuned-zp45w\" (UID: \"deaa6316-d948-414c-99b2-c4b07bbe68d8\") " pod="openshift-cluster-node-tuning-operator/tuned-zp45w" Jul 09 15:31:20.845962 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:31:20.845051 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/02be15c9-9800-4cae-9293-de04d5c845f8-konnectivity-ca\") pod \"konnectivity-agent-zrrxr\" (UID: \"02be15c9-9800-4cae-9293-de04d5c845f8\") " pod="kube-system/konnectivity-agent-zrrxr" Jul 09 15:31:20.846568 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:31:20.846546 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/deaa6316-d948-414c-99b2-c4b07bbe68d8-etc-tuned\") pod \"tuned-zp45w\" (UID: \"deaa6316-d948-414c-99b2-c4b07bbe68d8\") " pod="openshift-cluster-node-tuning-operator/tuned-zp45w" Jul 09 15:31:20.846755 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:31:20.846735 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/02be15c9-9800-4cae-9293-de04d5c845f8-agent-certs\") pod \"konnectivity-agent-zrrxr\" (UID: \"02be15c9-9800-4cae-9293-de04d5c845f8\") " pod="kube-system/konnectivity-agent-zrrxr" Jul 09 15:31:20.846938 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:31:20.846924 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/deaa6316-d948-414c-99b2-c4b07bbe68d8-tmp\") pod \"tuned-zp45w\" (UID: \"deaa6316-d948-414c-99b2-c4b07bbe68d8\") " pod="openshift-cluster-node-tuning-operator/tuned-zp45w" Jul 09 15:31:20.855246 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:31:20.855186 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9cw2h\" (UniqueName: \"kubernetes.io/projected/8e56f4e6-cbec-422f-9505-c7c6654159d9-kube-api-access-9cw2h\") pod \"network-metrics-daemon-8f2p4\" (UID: \"8e56f4e6-cbec-422f-9505-c7c6654159d9\") " pod="openshift-multus/network-metrics-daemon-8f2p4" Jul 09 15:31:20.855692 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:31:20.855670 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-s26wd\" (UniqueName: \"kubernetes.io/projected/9ad54b5e-c82e-480c-814f-e1ab55bf32fb-kube-api-access-s26wd\") pod \"node-resolver-b5c8h\" (UID: \"9ad54b5e-c82e-480c-814f-e1ab55bf32fb\") " pod="openshift-dns/node-resolver-b5c8h" Jul 09 15:31:20.855747 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:31:20.855726 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8pzhh\" (UniqueName: \"kubernetes.io/projected/1094a821-3b3e-46d8-ac90-64c7ae6e8cad-kube-api-access-8pzhh\") pod \"node-ca-trnjd\" (UID: \"1094a821-3b3e-46d8-ac90-64c7ae6e8cad\") " pod="openshift-image-registry/node-ca-trnjd" Jul 09 15:31:20.856001 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:31:20.855985 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-986n2\" (UniqueName: \"kubernetes.io/projected/deaa6316-d948-414c-99b2-c4b07bbe68d8-kube-api-access-986n2\") pod \"tuned-zp45w\" (UID: \"deaa6316-d948-414c-99b2-c4b07bbe68d8\") " pod="openshift-cluster-node-tuning-operator/tuned-zp45w" Jul 09 15:31:20.931629 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:31:20.931597 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-b7b9v" Jul 09 15:31:20.937433 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:31:20.937413 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vxr6x" Jul 09 15:31:20.939725 ip-10-0-133-183 kubenswrapper[2573]: W0709 15:31:20.939704 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddb22f88d_a953_488e_b384_67f991351f73.slice/crio-f85029fa5c5ec5092b9405015999277a2696a7e930a9629c368220eb58bd47e5 WatchSource:0}: Error finding container f85029fa5c5ec5092b9405015999277a2696a7e930a9629c368220eb58bd47e5: Status 404 returned error can't find the container with id f85029fa5c5ec5092b9405015999277a2696a7e930a9629c368220eb58bd47e5 Jul 09 15:31:20.944594 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:31:20.944572 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-g5qvc" Jul 09 15:31:20.944816 ip-10-0-133-183 kubenswrapper[2573]: W0709 15:31:20.944799 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod38e3cf1c_0139_47f3_a0f2_708c4e43b855.slice/crio-e92be106e632c2205d3b6759bb071f4002a4b17578e0b13d0ddbd83ae25b44f2 WatchSource:0}: Error finding container e92be106e632c2205d3b6759bb071f4002a4b17578e0b13d0ddbd83ae25b44f2: Status 404 returned error can't find the container with id e92be106e632c2205d3b6759bb071f4002a4b17578e0b13d0ddbd83ae25b44f2 Jul 09 15:31:20.947817 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:31:20.947789 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-v27h9" Jul 09 15:31:20.953861 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:31:20.953839 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-7zxwk" Jul 09 15:31:20.954846 ip-10-0-133-183 kubenswrapper[2573]: W0709 15:31:20.954809 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod85842275_e9c5_44af_8e99_1700a37602e3.slice/crio-ca85c768fe816beaa18ab69e414167f4b7ee2bbf85d2b79a37e5663642697030 WatchSource:0}: Error finding container ca85c768fe816beaa18ab69e414167f4b7ee2bbf85d2b79a37e5663642697030: Status 404 returned error can't find the container with id ca85c768fe816beaa18ab69e414167f4b7ee2bbf85d2b79a37e5663642697030 Jul 09 15:31:20.957923 ip-10-0-133-183 kubenswrapper[2573]: W0709 15:31:20.957904 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod86ea7a2d_fc63_4660_a266_93fe5360fe49.slice/crio-40f631b70f07af2b518bca9101920e5583ef13c08ec83e054e866553cbbfaab7 WatchSource:0}: Error finding container 40f631b70f07af2b518bca9101920e5583ef13c08ec83e054e866553cbbfaab7: Status 404 returned error can't find the container with id 40f631b70f07af2b518bca9101920e5583ef13c08ec83e054e866553cbbfaab7 Jul 09 15:31:20.958799 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:31:20.958750 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-zp45w" Jul 09 15:31:20.962616 ip-10-0-133-183 kubenswrapper[2573]: W0709 15:31:20.962593 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod07a170e7_eeff_4591_9a10_ecfe6a8e8100.slice/crio-442266c912f5758e9ee7e6a480a1c636237e2c607a3be35166e3a2b18ea5c7b3 WatchSource:0}: Error finding container 442266c912f5758e9ee7e6a480a1c636237e2c607a3be35166e3a2b18ea5c7b3: Status 404 returned error can't find the container with id 442266c912f5758e9ee7e6a480a1c636237e2c607a3be35166e3a2b18ea5c7b3 Jul 09 15:31:20.964470 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:31:20.964451 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-zrrxr" Jul 09 15:31:20.966605 ip-10-0-133-183 kubenswrapper[2573]: W0709 15:31:20.966578 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddeaa6316_d948_414c_99b2_c4b07bbe68d8.slice/crio-11b8af3563bb8311ee3c6d9c59ca92409074c8ee02352e8caefdb391216430a4 WatchSource:0}: Error finding container 11b8af3563bb8311ee3c6d9c59ca92409074c8ee02352e8caefdb391216430a4: Status 404 returned error can't find the container with id 11b8af3563bb8311ee3c6d9c59ca92409074c8ee02352e8caefdb391216430a4 Jul 09 15:31:20.969206 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:31:20.969191 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-b5c8h" Jul 09 15:31:20.971900 ip-10-0-133-183 kubenswrapper[2573]: W0709 15:31:20.971862 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod02be15c9_9800_4cae_9293_de04d5c845f8.slice/crio-de56f046373d9fdf957ebb2769fc338937e7c687a20bc0c27f72095c841c520a WatchSource:0}: Error finding container de56f046373d9fdf957ebb2769fc338937e7c687a20bc0c27f72095c841c520a: Status 404 returned error can't find the container with id de56f046373d9fdf957ebb2769fc338937e7c687a20bc0c27f72095c841c520a Jul 09 15:31:20.972444 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:31:20.972428 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-trnjd" Jul 09 15:31:20.978971 ip-10-0-133-183 kubenswrapper[2573]: W0709 15:31:20.978951 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9ad54b5e_c82e_480c_814f_e1ab55bf32fb.slice/crio-22f21c8c50db2183eb8f36253c897f89b5b88389a5d6a288ae7d4066ee0ccc57 WatchSource:0}: Error finding container 22f21c8c50db2183eb8f36253c897f89b5b88389a5d6a288ae7d4066ee0ccc57: Status 404 returned error can't find the container with id 22f21c8c50db2183eb8f36253c897f89b5b88389a5d6a288ae7d4066ee0ccc57 Jul 09 15:31:20.983058 ip-10-0-133-183 kubenswrapper[2573]: W0709 15:31:20.983038 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1094a821_3b3e_46d8_ac90_64c7ae6e8cad.slice/crio-a19a9c156362461a0a2fe35310448c31349c39429f74e37191de27338871eb62 WatchSource:0}: Error finding container a19a9c156362461a0a2fe35310448c31349c39429f74e37191de27338871eb62: Status 404 returned error can't find the container with id a19a9c156362461a0a2fe35310448c31349c39429f74e37191de27338871eb62 Jul 09 15:31:21.017448 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:31:21.017424 2573 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160" Jul 09 15:31:21.027379 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:31:21.027361 2573 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160" Jul 09 15:31:21.347306 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:31:21.347223 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/d9ea5934-356a-47e5-9fc6-1ae4f39f4e19-original-pull-secret\") pod \"global-pull-secret-syncer-h2cs6\" (UID: \"d9ea5934-356a-47e5-9fc6-1ae4f39f4e19\") " pod="kube-system/global-pull-secret-syncer-h2cs6" Jul 09 15:31:21.347478 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:31:21.347280 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8e56f4e6-cbec-422f-9505-c7c6654159d9-metrics-certs\") pod \"network-metrics-daemon-8f2p4\" (UID: \"8e56f4e6-cbec-422f-9505-c7c6654159d9\") " pod="openshift-multus/network-metrics-daemon-8f2p4" Jul 09 15:31:21.347478 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:31:21.347368 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l6q9n\" (UniqueName: \"kubernetes.io/projected/e8de13cc-b0e0-4a54-bd44-3ba73cbf2bf2-kube-api-access-l6q9n\") pod \"network-check-target-sdbbl\" (UID: \"e8de13cc-b0e0-4a54-bd44-3ba73cbf2bf2\") " pod="openshift-network-diagnostics/network-check-target-sdbbl" Jul 09 15:31:21.347478 ip-10-0-133-183 kubenswrapper[2573]: E0709 15:31:21.347419 2573 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 09 15:31:21.347638 ip-10-0-133-183 kubenswrapper[2573]: E0709 15:31:21.347510 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d9ea5934-356a-47e5-9fc6-1ae4f39f4e19-original-pull-secret podName:d9ea5934-356a-47e5-9fc6-1ae4f39f4e19 nodeName:}" failed. No retries permitted until 2026-07-09 15:31:22.347488435 +0000 UTC m=+4.068995297 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/d9ea5934-356a-47e5-9fc6-1ae4f39f4e19-original-pull-secret") pod "global-pull-secret-syncer-h2cs6" (UID: "d9ea5934-356a-47e5-9fc6-1ae4f39f4e19") : object "kube-system"/"original-pull-secret" not registered Jul 09 15:31:21.347638 ip-10-0-133-183 kubenswrapper[2573]: E0709 15:31:21.347516 2573 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 09 15:31:21.347638 ip-10-0-133-183 kubenswrapper[2573]: E0709 15:31:21.347534 2573 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 09 15:31:21.347638 ip-10-0-133-183 kubenswrapper[2573]: E0709 15:31:21.347546 2573 projected.go:196] Error preparing data for projected volume kube-api-access-l6q9n for pod openshift-network-diagnostics/network-check-target-sdbbl: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 09 15:31:21.347638 ip-10-0-133-183 kubenswrapper[2573]: E0709 15:31:21.347597 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/e8de13cc-b0e0-4a54-bd44-3ba73cbf2bf2-kube-api-access-l6q9n podName:e8de13cc-b0e0-4a54-bd44-3ba73cbf2bf2 nodeName:}" failed. No retries permitted until 2026-07-09 15:31:22.347580521 +0000 UTC m=+4.069087382 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-l6q9n" (UniqueName: "kubernetes.io/projected/e8de13cc-b0e0-4a54-bd44-3ba73cbf2bf2-kube-api-access-l6q9n") pod "network-check-target-sdbbl" (UID: "e8de13cc-b0e0-4a54-bd44-3ba73cbf2bf2") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 09 15:31:21.347864 ip-10-0-133-183 kubenswrapper[2573]: E0709 15:31:21.347664 2573 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 09 15:31:21.347864 ip-10-0-133-183 kubenswrapper[2573]: E0709 15:31:21.347702 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8e56f4e6-cbec-422f-9505-c7c6654159d9-metrics-certs podName:8e56f4e6-cbec-422f-9505-c7c6654159d9 nodeName:}" failed. No retries permitted until 2026-07-09 15:31:22.347690237 +0000 UTC m=+4.069197091 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/8e56f4e6-cbec-422f-9505-c7c6654159d9-metrics-certs") pod "network-metrics-daemon-8f2p4" (UID: "8e56f4e6-cbec-422f-9505-c7c6654159d9") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 09 15:31:21.675390 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:31:21.675219 2573 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-08 15:26:20 +0000 UTC" deadline="2028-01-15 20:32:57.226904345 +0000 UTC" Jul 09 15:31:21.675390 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:31:21.675285 2573 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13325h1m35.551623671s" Jul 09 15:31:21.805336 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:31:21.804778 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-h2cs6" Jul 09 15:31:21.805336 ip-10-0-133-183 kubenswrapper[2573]: E0709 15:31:21.804938 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-h2cs6" podUID="d9ea5934-356a-47e5-9fc6-1ae4f39f4e19" Jul 09 15:31:21.827891 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:31:21.827757 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-zrrxr" event={"ID":"02be15c9-9800-4cae-9293-de04d5c845f8","Type":"ContainerStarted","Data":"de56f046373d9fdf957ebb2769fc338937e7c687a20bc0c27f72095c841c520a"} Jul 09 15:31:21.831850 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:31:21.831799 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-g5qvc" event={"ID":"85842275-e9c5-44af-8e99-1700a37602e3","Type":"ContainerStarted","Data":"ca85c768fe816beaa18ab69e414167f4b7ee2bbf85d2b79a37e5663642697030"} Jul 09 15:31:21.836149 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:31:21.836114 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-trnjd" event={"ID":"1094a821-3b3e-46d8-ac90-64c7ae6e8cad","Type":"ContainerStarted","Data":"a19a9c156362461a0a2fe35310448c31349c39429f74e37191de27338871eb62"} Jul 09 15:31:21.842203 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:31:21.842165 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-zp45w" event={"ID":"deaa6316-d948-414c-99b2-c4b07bbe68d8","Type":"ContainerStarted","Data":"11b8af3563bb8311ee3c6d9c59ca92409074c8ee02352e8caefdb391216430a4"} Jul 09 15:31:21.846537 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:31:21.846501 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-7zxwk" event={"ID":"07a170e7-eeff-4591-9a10-ecfe6a8e8100","Type":"ContainerStarted","Data":"442266c912f5758e9ee7e6a480a1c636237e2c607a3be35166e3a2b18ea5c7b3"} Jul 09 15:31:21.850843 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:31:21.850790 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-v27h9" event={"ID":"86ea7a2d-fc63-4660-a266-93fe5360fe49","Type":"ContainerStarted","Data":"40f631b70f07af2b518bca9101920e5583ef13c08ec83e054e866553cbbfaab7"} Jul 09 15:31:21.860189 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:31:21.860153 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vxr6x" event={"ID":"38e3cf1c-0139-47f3-a0f2-708c4e43b855","Type":"ContainerStarted","Data":"e92be106e632c2205d3b6759bb071f4002a4b17578e0b13d0ddbd83ae25b44f2"} Jul 09 15:31:21.867319 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:31:21.867278 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-b7b9v" event={"ID":"db22f88d-a953-488e-b384-67f991351f73","Type":"ContainerStarted","Data":"f85029fa5c5ec5092b9405015999277a2696a7e930a9629c368220eb58bd47e5"} Jul 09 15:31:21.873993 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:31:21.873956 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-b5c8h" event={"ID":"9ad54b5e-c82e-480c-814f-e1ab55bf32fb","Type":"ContainerStarted","Data":"22f21c8c50db2183eb8f36253c897f89b5b88389a5d6a288ae7d4066ee0ccc57"} Jul 09 15:31:22.356099 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:31:22.356008 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8e56f4e6-cbec-422f-9505-c7c6654159d9-metrics-certs\") pod \"network-metrics-daemon-8f2p4\" (UID: \"8e56f4e6-cbec-422f-9505-c7c6654159d9\") " pod="openshift-multus/network-metrics-daemon-8f2p4" Jul 09 15:31:22.356099 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:31:22.356075 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l6q9n\" (UniqueName: \"kubernetes.io/projected/e8de13cc-b0e0-4a54-bd44-3ba73cbf2bf2-kube-api-access-l6q9n\") pod \"network-check-target-sdbbl\" (UID: \"e8de13cc-b0e0-4a54-bd44-3ba73cbf2bf2\") " pod="openshift-network-diagnostics/network-check-target-sdbbl" Jul 09 15:31:22.356342 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:31:22.356140 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/d9ea5934-356a-47e5-9fc6-1ae4f39f4e19-original-pull-secret\") pod \"global-pull-secret-syncer-h2cs6\" (UID: \"d9ea5934-356a-47e5-9fc6-1ae4f39f4e19\") " pod="kube-system/global-pull-secret-syncer-h2cs6" Jul 09 15:31:22.356342 ip-10-0-133-183 kubenswrapper[2573]: E0709 15:31:22.356257 2573 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 09 15:31:22.356342 ip-10-0-133-183 kubenswrapper[2573]: E0709 15:31:22.356332 2573 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 09 15:31:22.356485 ip-10-0-133-183 kubenswrapper[2573]: E0709 15:31:22.356346 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d9ea5934-356a-47e5-9fc6-1ae4f39f4e19-original-pull-secret podName:d9ea5934-356a-47e5-9fc6-1ae4f39f4e19 nodeName:}" failed. No retries permitted until 2026-07-09 15:31:24.35630135 +0000 UTC m=+6.077808216 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/d9ea5934-356a-47e5-9fc6-1ae4f39f4e19-original-pull-secret") pod "global-pull-secret-syncer-h2cs6" (UID: "d9ea5934-356a-47e5-9fc6-1ae4f39f4e19") : object "kube-system"/"original-pull-secret" not registered Jul 09 15:31:22.356485 ip-10-0-133-183 kubenswrapper[2573]: E0709 15:31:22.356374 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8e56f4e6-cbec-422f-9505-c7c6654159d9-metrics-certs podName:8e56f4e6-cbec-422f-9505-c7c6654159d9 nodeName:}" failed. No retries permitted until 2026-07-09 15:31:24.356363556 +0000 UTC m=+6.077870408 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/8e56f4e6-cbec-422f-9505-c7c6654159d9-metrics-certs") pod "network-metrics-daemon-8f2p4" (UID: "8e56f4e6-cbec-422f-9505-c7c6654159d9") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 09 15:31:22.356485 ip-10-0-133-183 kubenswrapper[2573]: E0709 15:31:22.356458 2573 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 09 15:31:22.356485 ip-10-0-133-183 kubenswrapper[2573]: E0709 15:31:22.356472 2573 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 09 15:31:22.356485 ip-10-0-133-183 kubenswrapper[2573]: E0709 15:31:22.356485 2573 projected.go:196] Error preparing data for projected volume kube-api-access-l6q9n for pod openshift-network-diagnostics/network-check-target-sdbbl: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 09 15:31:22.356750 ip-10-0-133-183 kubenswrapper[2573]: E0709 15:31:22.356516 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/e8de13cc-b0e0-4a54-bd44-3ba73cbf2bf2-kube-api-access-l6q9n podName:e8de13cc-b0e0-4a54-bd44-3ba73cbf2bf2 nodeName:}" failed. No retries permitted until 2026-07-09 15:31:24.356506283 +0000 UTC m=+6.078013135 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-l6q9n" (UniqueName: "kubernetes.io/projected/e8de13cc-b0e0-4a54-bd44-3ba73cbf2bf2-kube-api-access-l6q9n") pod "network-check-target-sdbbl" (UID: "e8de13cc-b0e0-4a54-bd44-3ba73cbf2bf2") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 09 15:31:22.675828 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:31:22.675732 2573 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-08 15:26:20 +0000 UTC" deadline="2028-03-14 13:06:18.138607543 +0000 UTC" Jul 09 15:31:22.675828 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:31:22.675775 2573 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14733h34m55.462836569s" Jul 09 15:31:22.745945 ip-10-0-133-183 kubenswrapper[2573]: W0709 15:31:22.745897 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfe0ed806b6a4ba1e258c73e86f255df7.slice/crio-6e6f6a1175e6b2777a2083fe4060b60e3b9826462ff749057882b43689b5e5a3 WatchSource:0}: Error finding container 6e6f6a1175e6b2777a2083fe4060b60e3b9826462ff749057882b43689b5e5a3: Status 404 returned error can't find the container with id 6e6f6a1175e6b2777a2083fe4060b60e3b9826462ff749057882b43689b5e5a3 Jul 09 15:31:22.807592 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:31:22.807559 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-sdbbl" Jul 09 15:31:22.807777 ip-10-0-133-183 kubenswrapper[2573]: E0709 15:31:22.807692 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-sdbbl" podUID="e8de13cc-b0e0-4a54-bd44-3ba73cbf2bf2" Jul 09 15:31:22.808141 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:31:22.808120 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-8f2p4" Jul 09 15:31:22.808244 ip-10-0-133-183 kubenswrapper[2573]: E0709 15:31:22.808230 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-8f2p4" podUID="8e56f4e6-cbec-422f-9505-c7c6654159d9" Jul 09 15:31:22.880423 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:31:22.880338 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-183.ec2.internal" event={"ID":"fe0ed806b6a4ba1e258c73e86f255df7","Type":"ContainerStarted","Data":"6e6f6a1175e6b2777a2083fe4060b60e3b9826462ff749057882b43689b5e5a3"} Jul 09 15:31:23.805843 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:31:23.805804 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-h2cs6" Jul 09 15:31:23.806350 ip-10-0-133-183 kubenswrapper[2573]: E0709 15:31:23.805969 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-h2cs6" podUID="d9ea5934-356a-47e5-9fc6-1ae4f39f4e19" Jul 09 15:31:24.372648 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:31:24.372604 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/d9ea5934-356a-47e5-9fc6-1ae4f39f4e19-original-pull-secret\") pod \"global-pull-secret-syncer-h2cs6\" (UID: \"d9ea5934-356a-47e5-9fc6-1ae4f39f4e19\") " pod="kube-system/global-pull-secret-syncer-h2cs6" Jul 09 15:31:24.372854 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:31:24.372664 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8e56f4e6-cbec-422f-9505-c7c6654159d9-metrics-certs\") pod \"network-metrics-daemon-8f2p4\" (UID: \"8e56f4e6-cbec-422f-9505-c7c6654159d9\") " pod="openshift-multus/network-metrics-daemon-8f2p4" Jul 09 15:31:24.372854 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:31:24.372698 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l6q9n\" (UniqueName: \"kubernetes.io/projected/e8de13cc-b0e0-4a54-bd44-3ba73cbf2bf2-kube-api-access-l6q9n\") pod \"network-check-target-sdbbl\" (UID: \"e8de13cc-b0e0-4a54-bd44-3ba73cbf2bf2\") " pod="openshift-network-diagnostics/network-check-target-sdbbl" Jul 09 15:31:24.372854 ip-10-0-133-183 kubenswrapper[2573]: E0709 15:31:24.372806 2573 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 09 15:31:24.372854 ip-10-0-133-183 kubenswrapper[2573]: E0709 15:31:24.372832 2573 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 09 15:31:24.372854 ip-10-0-133-183 kubenswrapper[2573]: E0709 15:31:24.372848 2573 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 09 15:31:24.373138 ip-10-0-133-183 kubenswrapper[2573]: E0709 15:31:24.372863 2573 projected.go:196] Error preparing data for projected volume kube-api-access-l6q9n for pod openshift-network-diagnostics/network-check-target-sdbbl: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 09 15:31:24.373138 ip-10-0-133-183 kubenswrapper[2573]: E0709 15:31:24.372911 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d9ea5934-356a-47e5-9fc6-1ae4f39f4e19-original-pull-secret podName:d9ea5934-356a-47e5-9fc6-1ae4f39f4e19 nodeName:}" failed. No retries permitted until 2026-07-09 15:31:28.372870335 +0000 UTC m=+10.094377185 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/d9ea5934-356a-47e5-9fc6-1ae4f39f4e19-original-pull-secret") pod "global-pull-secret-syncer-h2cs6" (UID: "d9ea5934-356a-47e5-9fc6-1ae4f39f4e19") : object "kube-system"/"original-pull-secret" not registered Jul 09 15:31:24.373138 ip-10-0-133-183 kubenswrapper[2573]: E0709 15:31:24.372925 2573 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 09 15:31:24.373138 ip-10-0-133-183 kubenswrapper[2573]: E0709 15:31:24.372937 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/e8de13cc-b0e0-4a54-bd44-3ba73cbf2bf2-kube-api-access-l6q9n podName:e8de13cc-b0e0-4a54-bd44-3ba73cbf2bf2 nodeName:}" failed. No retries permitted until 2026-07-09 15:31:28.372925738 +0000 UTC m=+10.094432585 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-l6q9n" (UniqueName: "kubernetes.io/projected/e8de13cc-b0e0-4a54-bd44-3ba73cbf2bf2-kube-api-access-l6q9n") pod "network-check-target-sdbbl" (UID: "e8de13cc-b0e0-4a54-bd44-3ba73cbf2bf2") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 09 15:31:24.373138 ip-10-0-133-183 kubenswrapper[2573]: E0709 15:31:24.372978 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8e56f4e6-cbec-422f-9505-c7c6654159d9-metrics-certs podName:8e56f4e6-cbec-422f-9505-c7c6654159d9 nodeName:}" failed. No retries permitted until 2026-07-09 15:31:28.372962435 +0000 UTC m=+10.094469281 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/8e56f4e6-cbec-422f-9505-c7c6654159d9-metrics-certs") pod "network-metrics-daemon-8f2p4" (UID: "8e56f4e6-cbec-422f-9505-c7c6654159d9") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 09 15:31:24.805920 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:31:24.805778 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-sdbbl" Jul 09 15:31:24.806379 ip-10-0-133-183 kubenswrapper[2573]: E0709 15:31:24.805936 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-sdbbl" podUID="e8de13cc-b0e0-4a54-bd44-3ba73cbf2bf2" Jul 09 15:31:24.806379 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:31:24.806285 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-8f2p4" Jul 09 15:31:24.806507 ip-10-0-133-183 kubenswrapper[2573]: E0709 15:31:24.806374 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-8f2p4" podUID="8e56f4e6-cbec-422f-9505-c7c6654159d9" Jul 09 15:31:25.805681 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:31:25.805632 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-h2cs6" Jul 09 15:31:25.805888 ip-10-0-133-183 kubenswrapper[2573]: E0709 15:31:25.805781 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-h2cs6" podUID="d9ea5934-356a-47e5-9fc6-1ae4f39f4e19" Jul 09 15:31:26.805585 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:31:26.805247 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-sdbbl" Jul 09 15:31:26.806046 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:31:26.805252 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-8f2p4" Jul 09 15:31:26.806046 ip-10-0-133-183 kubenswrapper[2573]: E0709 15:31:26.805630 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-sdbbl" podUID="e8de13cc-b0e0-4a54-bd44-3ba73cbf2bf2" Jul 09 15:31:26.806046 ip-10-0-133-183 kubenswrapper[2573]: E0709 15:31:26.805793 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-8f2p4" podUID="8e56f4e6-cbec-422f-9505-c7c6654159d9" Jul 09 15:31:27.805371 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:31:27.805334 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-h2cs6" Jul 09 15:31:27.805536 ip-10-0-133-183 kubenswrapper[2573]: E0709 15:31:27.805480 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-h2cs6" podUID="d9ea5934-356a-47e5-9fc6-1ae4f39f4e19" Jul 09 15:31:28.407304 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:31:28.407254 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/d9ea5934-356a-47e5-9fc6-1ae4f39f4e19-original-pull-secret\") pod \"global-pull-secret-syncer-h2cs6\" (UID: \"d9ea5934-356a-47e5-9fc6-1ae4f39f4e19\") " pod="kube-system/global-pull-secret-syncer-h2cs6" Jul 09 15:31:28.407798 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:31:28.407321 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8e56f4e6-cbec-422f-9505-c7c6654159d9-metrics-certs\") pod \"network-metrics-daemon-8f2p4\" (UID: \"8e56f4e6-cbec-422f-9505-c7c6654159d9\") " pod="openshift-multus/network-metrics-daemon-8f2p4" Jul 09 15:31:28.407798 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:31:28.407359 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l6q9n\" (UniqueName: \"kubernetes.io/projected/e8de13cc-b0e0-4a54-bd44-3ba73cbf2bf2-kube-api-access-l6q9n\") pod \"network-check-target-sdbbl\" (UID: \"e8de13cc-b0e0-4a54-bd44-3ba73cbf2bf2\") " pod="openshift-network-diagnostics/network-check-target-sdbbl" Jul 09 15:31:28.407798 ip-10-0-133-183 kubenswrapper[2573]: E0709 15:31:28.407415 2573 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 09 15:31:28.407798 ip-10-0-133-183 kubenswrapper[2573]: E0709 15:31:28.407436 2573 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 09 15:31:28.407798 ip-10-0-133-183 kubenswrapper[2573]: E0709 15:31:28.407500 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d9ea5934-356a-47e5-9fc6-1ae4f39f4e19-original-pull-secret podName:d9ea5934-356a-47e5-9fc6-1ae4f39f4e19 nodeName:}" failed. No retries permitted until 2026-07-09 15:31:36.407479999 +0000 UTC m=+18.128986854 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/d9ea5934-356a-47e5-9fc6-1ae4f39f4e19-original-pull-secret") pod "global-pull-secret-syncer-h2cs6" (UID: "d9ea5934-356a-47e5-9fc6-1ae4f39f4e19") : object "kube-system"/"original-pull-secret" not registered Jul 09 15:31:28.407798 ip-10-0-133-183 kubenswrapper[2573]: E0709 15:31:28.407518 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8e56f4e6-cbec-422f-9505-c7c6654159d9-metrics-certs podName:8e56f4e6-cbec-422f-9505-c7c6654159d9 nodeName:}" failed. No retries permitted until 2026-07-09 15:31:36.407509665 +0000 UTC m=+18.129016517 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/8e56f4e6-cbec-422f-9505-c7c6654159d9-metrics-certs") pod "network-metrics-daemon-8f2p4" (UID: "8e56f4e6-cbec-422f-9505-c7c6654159d9") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 09 15:31:28.407798 ip-10-0-133-183 kubenswrapper[2573]: E0709 15:31:28.407524 2573 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 09 15:31:28.407798 ip-10-0-133-183 kubenswrapper[2573]: E0709 15:31:28.407543 2573 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 09 15:31:28.407798 ip-10-0-133-183 kubenswrapper[2573]: E0709 15:31:28.407556 2573 projected.go:196] Error preparing data for projected volume kube-api-access-l6q9n for pod openshift-network-diagnostics/network-check-target-sdbbl: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 09 15:31:28.407798 ip-10-0-133-183 kubenswrapper[2573]: E0709 15:31:28.407609 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/e8de13cc-b0e0-4a54-bd44-3ba73cbf2bf2-kube-api-access-l6q9n podName:e8de13cc-b0e0-4a54-bd44-3ba73cbf2bf2 nodeName:}" failed. No retries permitted until 2026-07-09 15:31:36.407592265 +0000 UTC m=+18.129099171 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-l6q9n" (UniqueName: "kubernetes.io/projected/e8de13cc-b0e0-4a54-bd44-3ba73cbf2bf2-kube-api-access-l6q9n") pod "network-check-target-sdbbl" (UID: "e8de13cc-b0e0-4a54-bd44-3ba73cbf2bf2") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 09 15:31:28.806574 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:31:28.806036 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-8f2p4" Jul 09 15:31:28.806574 ip-10-0-133-183 kubenswrapper[2573]: E0709 15:31:28.806166 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-8f2p4" podUID="8e56f4e6-cbec-422f-9505-c7c6654159d9" Jul 09 15:31:28.806574 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:31:28.806500 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-sdbbl" Jul 09 15:31:28.806841 ip-10-0-133-183 kubenswrapper[2573]: E0709 15:31:28.806618 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-sdbbl" podUID="e8de13cc-b0e0-4a54-bd44-3ba73cbf2bf2" Jul 09 15:31:29.805705 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:31:29.805660 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-h2cs6" Jul 09 15:31:29.806242 ip-10-0-133-183 kubenswrapper[2573]: E0709 15:31:29.805806 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-h2cs6" podUID="d9ea5934-356a-47e5-9fc6-1ae4f39f4e19" Jul 09 15:31:29.900359 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:31:29.900315 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vxr6x" event={"ID":"38e3cf1c-0139-47f3-a0f2-708c4e43b855","Type":"ContainerStarted","Data":"05f39cb62c7f228c7d12eefffe7e0b4bfebb9224d6af15599c3ae04c6755d03d"} Jul 09 15:31:29.903283 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:31:29.903246 2573 generic.go:356] "Generic (PLEG): container finished" podID="fe0ed806b6a4ba1e258c73e86f255df7" containerID="2057fb5c456bc81474fcf8d94edd8c5e8d804eceb3b0992ffaf1309ec87aeada" exitCode=0 Jul 09 15:31:29.903440 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:31:29.903347 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-183.ec2.internal" event={"ID":"fe0ed806b6a4ba1e258c73e86f255df7","Type":"ContainerDied","Data":"2057fb5c456bc81474fcf8d94edd8c5e8d804eceb3b0992ffaf1309ec87aeada"} Jul 09 15:31:29.905507 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:31:29.905475 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-b5c8h" event={"ID":"9ad54b5e-c82e-480c-814f-e1ab55bf32fb","Type":"ContainerStarted","Data":"dbd678b9dab99f3f781571a8bdd2ac7ea9ec182d80474c6885b9075933a542b3"} Jul 09 15:31:29.907972 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:31:29.907897 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-zrrxr" event={"ID":"02be15c9-9800-4cae-9293-de04d5c845f8","Type":"ContainerStarted","Data":"b4685f1e929dce7270838d55ee3aeed20a75e23be770295fed268545ea69e226"} Jul 09 15:31:29.910255 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:31:29.910208 2573 generic.go:356] "Generic (PLEG): container finished" podID="85842275-e9c5-44af-8e99-1700a37602e3" containerID="0638c752659f07e68521b12f8123417a538a5d20e631e4c58a607a0ff00dd752" exitCode=0 Jul 09 15:31:29.910377 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:31:29.910291 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-g5qvc" event={"ID":"85842275-e9c5-44af-8e99-1700a37602e3","Type":"ContainerDied","Data":"0638c752659f07e68521b12f8123417a538a5d20e631e4c58a607a0ff00dd752"} Jul 09 15:31:29.912009 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:31:29.911944 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-trnjd" event={"ID":"1094a821-3b3e-46d8-ac90-64c7ae6e8cad","Type":"ContainerStarted","Data":"0a0a5565cf2347c031b571ee09297d1cd30f46cd2d48299a230109997de1fc65"} Jul 09 15:31:29.941742 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:31:29.941684 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-trnjd" podStartSLOduration=3.985633936 podStartE2EDuration="11.941665523s" totalImagesPullingTime="7.956031587s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-09 15:31:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 15:31:29.941256256 +0000 UTC m=+11.662763151" watchObservedRunningTime="2026-07-09 15:31:29.941665523 +0000 UTC m=+11.663172393" Jul 09 15:31:29.956422 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:31:29.956371 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-zrrxr" podStartSLOduration=3.988229104 podStartE2EDuration="11.956353915s" totalImagesPullingTime="7.968124811s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 15:31:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 15:31:29.956281111 +0000 UTC m=+11.677787981" watchObservedRunningTime="2026-07-09 15:31:29.956353915 +0000 UTC m=+11.677860784" Jul 09 15:31:29.972159 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:31:29.972107 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-b5c8h" podStartSLOduration=3.9679532 podStartE2EDuration="11.972089405s" totalImagesPullingTime="8.004136205s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-09 15:31:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 15:31:29.97203841 +0000 UTC m=+11.693545280" watchObservedRunningTime="2026-07-09 15:31:29.972089405 +0000 UTC m=+11.693596273" Jul 09 15:31:30.805471 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:31:30.805434 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-8f2p4" Jul 09 15:31:30.805656 ip-10-0-133-183 kubenswrapper[2573]: E0709 15:31:30.805584 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-8f2p4" podUID="8e56f4e6-cbec-422f-9505-c7c6654159d9" Jul 09 15:31:30.805656 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:31:30.805612 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-sdbbl" Jul 09 15:31:30.806089 ip-10-0-133-183 kubenswrapper[2573]: E0709 15:31:30.805736 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-sdbbl" podUID="e8de13cc-b0e0-4a54-bd44-3ba73cbf2bf2" Jul 09 15:31:30.917741 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:31:30.917648 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-7zxwk" event={"ID":"07a170e7-eeff-4591-9a10-ecfe6a8e8100","Type":"ContainerStarted","Data":"813cbf9972634b4fe7ea6382a7fe20bb173a4ab156bb0e5a54a6ee4c4c6c1855"} Jul 09 15:31:30.937545 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:31:30.937449 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-7zxwk" podStartSLOduration=4.958034876 podStartE2EDuration="12.937426507s" totalImagesPullingTime="7.979391631s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-09 15:31:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 15:31:30.936651017 +0000 UTC m=+12.658157914" watchObservedRunningTime="2026-07-09 15:31:30.937426507 +0000 UTC m=+12.658933381" Jul 09 15:31:31.805424 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:31:31.805378 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-h2cs6" Jul 09 15:31:31.805612 ip-10-0-133-183 kubenswrapper[2573]: E0709 15:31:31.805525 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-h2cs6" podUID="d9ea5934-356a-47e5-9fc6-1ae4f39f4e19" Jul 09 15:31:32.596249 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:31:32.596201 2573 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-zrrxr" Jul 09 15:31:32.597091 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:31:32.596915 2573 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-zrrxr" Jul 09 15:31:32.805491 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:31:32.805239 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-8f2p4" Jul 09 15:31:32.805667 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:31:32.805239 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-sdbbl" Jul 09 15:31:32.805667 ip-10-0-133-183 kubenswrapper[2573]: E0709 15:31:32.805624 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-8f2p4" podUID="8e56f4e6-cbec-422f-9505-c7c6654159d9" Jul 09 15:31:32.805804 ip-10-0-133-183 kubenswrapper[2573]: E0709 15:31:32.805674 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-sdbbl" podUID="e8de13cc-b0e0-4a54-bd44-3ba73cbf2bf2" Jul 09 15:31:33.805479 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:31:33.805437 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-h2cs6" Jul 09 15:31:33.805999 ip-10-0-133-183 kubenswrapper[2573]: E0709 15:31:33.805564 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-h2cs6" podUID="d9ea5934-356a-47e5-9fc6-1ae4f39f4e19" Jul 09 15:31:34.805449 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:31:34.805412 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-sdbbl" Jul 09 15:31:34.805651 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:31:34.805421 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-8f2p4" Jul 09 15:31:34.805651 ip-10-0-133-183 kubenswrapper[2573]: E0709 15:31:34.805537 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-sdbbl" podUID="e8de13cc-b0e0-4a54-bd44-3ba73cbf2bf2" Jul 09 15:31:34.806149 ip-10-0-133-183 kubenswrapper[2573]: E0709 15:31:34.805650 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-8f2p4" podUID="8e56f4e6-cbec-422f-9505-c7c6654159d9" Jul 09 15:31:35.805132 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:31:35.805102 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-h2cs6" Jul 09 15:31:35.805369 ip-10-0-133-183 kubenswrapper[2573]: E0709 15:31:35.805213 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-h2cs6" podUID="d9ea5934-356a-47e5-9fc6-1ae4f39f4e19" Jul 09 15:31:36.470637 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:31:36.470605 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/d9ea5934-356a-47e5-9fc6-1ae4f39f4e19-original-pull-secret\") pod \"global-pull-secret-syncer-h2cs6\" (UID: \"d9ea5934-356a-47e5-9fc6-1ae4f39f4e19\") " pod="kube-system/global-pull-secret-syncer-h2cs6" Jul 09 15:31:36.471189 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:31:36.470646 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8e56f4e6-cbec-422f-9505-c7c6654159d9-metrics-certs\") pod \"network-metrics-daemon-8f2p4\" (UID: \"8e56f4e6-cbec-422f-9505-c7c6654159d9\") " pod="openshift-multus/network-metrics-daemon-8f2p4" Jul 09 15:31:36.471189 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:31:36.470671 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l6q9n\" (UniqueName: \"kubernetes.io/projected/e8de13cc-b0e0-4a54-bd44-3ba73cbf2bf2-kube-api-access-l6q9n\") pod \"network-check-target-sdbbl\" (UID: \"e8de13cc-b0e0-4a54-bd44-3ba73cbf2bf2\") " pod="openshift-network-diagnostics/network-check-target-sdbbl" Jul 09 15:31:36.471189 ip-10-0-133-183 kubenswrapper[2573]: E0709 15:31:36.470806 2573 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 09 15:31:36.471189 ip-10-0-133-183 kubenswrapper[2573]: E0709 15:31:36.470833 2573 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 09 15:31:36.471189 ip-10-0-133-183 kubenswrapper[2573]: E0709 15:31:36.470850 2573 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 09 15:31:36.471189 ip-10-0-133-183 kubenswrapper[2573]: E0709 15:31:36.470859 2573 projected.go:196] Error preparing data for projected volume kube-api-access-l6q9n for pod openshift-network-diagnostics/network-check-target-sdbbl: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 09 15:31:36.471189 ip-10-0-133-183 kubenswrapper[2573]: E0709 15:31:36.470871 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d9ea5934-356a-47e5-9fc6-1ae4f39f4e19-original-pull-secret podName:d9ea5934-356a-47e5-9fc6-1ae4f39f4e19 nodeName:}" failed. No retries permitted until 2026-07-09 15:31:52.470854489 +0000 UTC m=+34.192361336 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/d9ea5934-356a-47e5-9fc6-1ae4f39f4e19-original-pull-secret") pod "global-pull-secret-syncer-h2cs6" (UID: "d9ea5934-356a-47e5-9fc6-1ae4f39f4e19") : object "kube-system"/"original-pull-secret" not registered Jul 09 15:31:36.471189 ip-10-0-133-183 kubenswrapper[2573]: E0709 15:31:36.470810 2573 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 09 15:31:36.471189 ip-10-0-133-183 kubenswrapper[2573]: E0709 15:31:36.470918 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/e8de13cc-b0e0-4a54-bd44-3ba73cbf2bf2-kube-api-access-l6q9n podName:e8de13cc-b0e0-4a54-bd44-3ba73cbf2bf2 nodeName:}" failed. No retries permitted until 2026-07-09 15:31:52.470901643 +0000 UTC m=+34.192408492 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-l6q9n" (UniqueName: "kubernetes.io/projected/e8de13cc-b0e0-4a54-bd44-3ba73cbf2bf2-kube-api-access-l6q9n") pod "network-check-target-sdbbl" (UID: "e8de13cc-b0e0-4a54-bd44-3ba73cbf2bf2") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 09 15:31:36.471189 ip-10-0-133-183 kubenswrapper[2573]: E0709 15:31:36.470945 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8e56f4e6-cbec-422f-9505-c7c6654159d9-metrics-certs podName:8e56f4e6-cbec-422f-9505-c7c6654159d9 nodeName:}" failed. No retries permitted until 2026-07-09 15:31:52.470933072 +0000 UTC m=+34.192439928 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/8e56f4e6-cbec-422f-9505-c7c6654159d9-metrics-certs") pod "network-metrics-daemon-8f2p4" (UID: "8e56f4e6-cbec-422f-9505-c7c6654159d9") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 09 15:31:36.805367 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:31:36.805284 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-8f2p4" Jul 09 15:31:36.805545 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:31:36.805284 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-sdbbl" Jul 09 15:31:36.805545 ip-10-0-133-183 kubenswrapper[2573]: E0709 15:31:36.805425 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-8f2p4" podUID="8e56f4e6-cbec-422f-9505-c7c6654159d9" Jul 09 15:31:36.805545 ip-10-0-133-183 kubenswrapper[2573]: E0709 15:31:36.805511 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-sdbbl" podUID="e8de13cc-b0e0-4a54-bd44-3ba73cbf2bf2" Jul 09 15:31:37.805544 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:31:37.805501 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-h2cs6" Jul 09 15:31:37.806102 ip-10-0-133-183 kubenswrapper[2573]: E0709 15:31:37.805647 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-h2cs6" podUID="d9ea5934-356a-47e5-9fc6-1ae4f39f4e19" Jul 09 15:31:38.131467 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:31:38.131354 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-zrrxr" Jul 09 15:31:38.131619 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:31:38.131527 2573 prober_manager.go:312] "Failed to trigger a manual run" probe="Readiness" Jul 09 15:31:38.132031 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:31:38.132014 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-zrrxr" Jul 09 15:31:38.806692 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:31:38.806656 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-sdbbl" Jul 09 15:31:38.807161 ip-10-0-133-183 kubenswrapper[2573]: E0709 15:31:38.806777 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-sdbbl" podUID="e8de13cc-b0e0-4a54-bd44-3ba73cbf2bf2" Jul 09 15:31:38.807161 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:31:38.806832 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-8f2p4" Jul 09 15:31:38.807161 ip-10-0-133-183 kubenswrapper[2573]: E0709 15:31:38.806968 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-8f2p4" podUID="8e56f4e6-cbec-422f-9505-c7c6654159d9" Jul 09 15:31:39.805643 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:31:39.805599 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-h2cs6" Jul 09 15:31:39.805817 ip-10-0-133-183 kubenswrapper[2573]: E0709 15:31:39.805748 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-h2cs6" podUID="d9ea5934-356a-47e5-9fc6-1ae4f39f4e19" Jul 09 15:31:40.805765 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:31:40.805727 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-8f2p4" Jul 09 15:31:40.806441 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:31:40.805740 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-sdbbl" Jul 09 15:31:40.806441 ip-10-0-133-183 kubenswrapper[2573]: E0709 15:31:40.805910 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-8f2p4" podUID="8e56f4e6-cbec-422f-9505-c7c6654159d9" Jul 09 15:31:40.806441 ip-10-0-133-183 kubenswrapper[2573]: E0709 15:31:40.805953 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-sdbbl" podUID="e8de13cc-b0e0-4a54-bd44-3ba73cbf2bf2" Jul 09 15:31:41.805027 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:31:41.804994 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-h2cs6" Jul 09 15:31:41.805163 ip-10-0-133-183 kubenswrapper[2573]: E0709 15:31:41.805091 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-h2cs6" podUID="d9ea5934-356a-47e5-9fc6-1ae4f39f4e19" Jul 09 15:31:42.805409 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:31:42.805373 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-sdbbl" Jul 09 15:31:42.805913 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:31:42.805476 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-8f2p4" Jul 09 15:31:42.805913 ip-10-0-133-183 kubenswrapper[2573]: E0709 15:31:42.805530 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-sdbbl" podUID="e8de13cc-b0e0-4a54-bd44-3ba73cbf2bf2" Jul 09 15:31:42.805913 ip-10-0-133-183 kubenswrapper[2573]: E0709 15:31:42.805583 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-8f2p4" podUID="8e56f4e6-cbec-422f-9505-c7c6654159d9" Jul 09 15:31:43.807764 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:31:43.806277 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-h2cs6" Jul 09 15:31:43.807764 ip-10-0-133-183 kubenswrapper[2573]: E0709 15:31:43.806655 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-h2cs6" podUID="d9ea5934-356a-47e5-9fc6-1ae4f39f4e19" Jul 09 15:31:43.861001 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:31:43.860948 2573 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock" Jul 09 15:31:43.946203 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:31:43.946165 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-133-183.ec2.internal" event={"ID":"06227558853f1ac07df9b5c4930c43af","Type":"ContainerStarted","Data":"5b848842993a14e2839a0b46dfc6f6658d6558d306f33929e0b3593f2b6a6647"} Jul 09 15:31:43.947616 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:31:43.947589 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-zp45w" event={"ID":"deaa6316-d948-414c-99b2-c4b07bbe68d8","Type":"ContainerStarted","Data":"5db8aa24efbf02eeb0c35c12a7441e55a0f5bb6152b343be5296c3e6cbc54fd8"} Jul 09 15:31:43.949094 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:31:43.949071 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-v27h9" event={"ID":"86ea7a2d-fc63-4660-a266-93fe5360fe49","Type":"ContainerStarted","Data":"d528b94f4a48f259d337c7d0e84764c2c84e4147106d5ee25a10c98450dc635f"} Jul 09 15:31:43.950973 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:31:43.950952 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vxr6x" event={"ID":"38e3cf1c-0139-47f3-a0f2-708c4e43b855","Type":"ContainerStarted","Data":"fddcdb3e562d8e52da02a4f89eb6f548ea9c4371bbbcd2523d617070934c3d79"} Jul 09 15:31:43.952422 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:31:43.952407 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-b7b9v_db22f88d-a953-488e-b384-67f991351f73/ovn-acl-logging/0.log" Jul 09 15:31:43.952677 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:31:43.952654 2573 generic.go:356] "Generic (PLEG): container finished" podID="db22f88d-a953-488e-b384-67f991351f73" containerID="19eeface74077b327e95767180802bf9a2fdb5593bada3e61556a95703c87f8d" exitCode=1 Jul 09 15:31:43.952782 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:31:43.952681 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-b7b9v" event={"ID":"db22f88d-a953-488e-b384-67f991351f73","Type":"ContainerDied","Data":"19eeface74077b327e95767180802bf9a2fdb5593bada3e61556a95703c87f8d"} Jul 09 15:31:43.952782 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:31:43.952764 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-b7b9v" event={"ID":"db22f88d-a953-488e-b384-67f991351f73","Type":"ContainerStarted","Data":"968d440e2eb8439491a1e782e97ffe24e1457ac6e90e2979293fcd85b6e37ea9"} Jul 09 15:31:43.954457 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:31:43.954433 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-183.ec2.internal" event={"ID":"fe0ed806b6a4ba1e258c73e86f255df7","Type":"ContainerStarted","Data":"f6cf6150c52b5a53fc14c825560b06089f102ba9963b4841d67c7e198a38e3f7"} Jul 09 15:31:43.956057 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:31:43.956037 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-g5qvc" event={"ID":"85842275-e9c5-44af-8e99-1700a37602e3","Type":"ContainerStarted","Data":"de7e3f1650298e117adffc7e7b32745d9d9b5babf7c50c7e9c466ccb13e4a3e9"} Jul 09 15:31:43.962749 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:31:43.962706 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-133-183.ec2.internal" podStartSLOduration=23.962688814 podStartE2EDuration="23.962688814s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-09 15:31:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 15:31:43.96245335 +0000 UTC m=+25.683960220" watchObservedRunningTime="2026-07-09 15:31:43.962688814 +0000 UTC m=+25.684195685" Jul 09 15:31:43.981917 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:31:43.981852 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-zp45w" podStartSLOduration=3.371672779 podStartE2EDuration="25.981831937s" totalImagesPullingTime="22.610159158s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-09 15:31:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 15:31:43.981822263 +0000 UTC m=+25.703329131" watchObservedRunningTime="2026-07-09 15:31:43.981831937 +0000 UTC m=+25.703338809" Jul 09 15:31:43.997912 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:31:43.997847 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-183.ec2.internal" podStartSLOduration=23.9978306 podStartE2EDuration="23.9978306s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-09 15:31:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 15:31:43.997642337 +0000 UTC m=+25.719149233" watchObservedRunningTime="2026-07-09 15:31:43.9978306 +0000 UTC m=+25.719337469" Jul 09 15:31:44.042371 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:31:44.042326 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-v27h9" podStartSLOduration=3.422909741 podStartE2EDuration="26.0423093s" totalImagesPullingTime="22.619399559s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-09 15:31:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 15:31:44.041898137 +0000 UTC m=+25.763405023" watchObservedRunningTime="2026-07-09 15:31:44.0423093 +0000 UTC m=+25.763816167" Jul 09 15:31:44.711397 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:31:44.711096 2573 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-07-09T15:31:43.860970376Z","UUID":"a1d1cbfd-e76d-4bad-b398-7aa945be7825","Handler":null,"Name":"","Endpoint":""} Jul 09 15:31:44.712552 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:31:44.712530 2573 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0 Jul 09 15:31:44.712677 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:31:44.712567 2573 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock Jul 09 15:31:44.805122 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:31:44.805087 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-sdbbl" Jul 09 15:31:44.805122 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:31:44.805109 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-8f2p4" Jul 09 15:31:44.805375 ip-10-0-133-183 kubenswrapper[2573]: E0709 15:31:44.805223 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-sdbbl" podUID="e8de13cc-b0e0-4a54-bd44-3ba73cbf2bf2" Jul 09 15:31:44.805375 ip-10-0-133-183 kubenswrapper[2573]: E0709 15:31:44.805344 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-8f2p4" podUID="8e56f4e6-cbec-422f-9505-c7c6654159d9" Jul 09 15:31:44.962334 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:31:44.962267 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-b7b9v_db22f88d-a953-488e-b384-67f991351f73/ovn-acl-logging/0.log" Jul 09 15:31:44.962726 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:31:44.962682 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-b7b9v" event={"ID":"db22f88d-a953-488e-b384-67f991351f73","Type":"ContainerStarted","Data":"ad554d42a03fea3078e12c62b2f3619d5514e3b601b4fb4ac17744357b5b9a30"} Jul 09 15:31:44.962779 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:31:44.962765 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-b7b9v" event={"ID":"db22f88d-a953-488e-b384-67f991351f73","Type":"ContainerStarted","Data":"73846830f7a0935e6bcf7c9e53ceea0625b962419a9e7421b63a367afa2de3d4"} Jul 09 15:31:44.962815 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:31:44.962806 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-b7b9v" event={"ID":"db22f88d-a953-488e-b384-67f991351f73","Type":"ContainerStarted","Data":"4fbdd1f14758f63f0baa6e813bb06a4504e790f358ef7db21017b6dd049c6377"} Jul 09 15:31:44.962863 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:31:44.962837 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-b7b9v" event={"ID":"db22f88d-a953-488e-b384-67f991351f73","Type":"ContainerStarted","Data":"51a95227e65b895a2c260a1639f0daad1a3acdef97c6cbeaa405dd5ca4619ce9"} Jul 09 15:31:44.964266 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:31:44.964243 2573 generic.go:356] "Generic (PLEG): container finished" podID="85842275-e9c5-44af-8e99-1700a37602e3" containerID="de7e3f1650298e117adffc7e7b32745d9d9b5babf7c50c7e9c466ccb13e4a3e9" exitCode=0 Jul 09 15:31:44.964381 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:31:44.964349 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-g5qvc" event={"ID":"85842275-e9c5-44af-8e99-1700a37602e3","Type":"ContainerDied","Data":"de7e3f1650298e117adffc7e7b32745d9d9b5babf7c50c7e9c466ccb13e4a3e9"} Jul 09 15:31:45.805326 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:31:45.805236 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-h2cs6" Jul 09 15:31:45.805551 ip-10-0-133-183 kubenswrapper[2573]: E0709 15:31:45.805363 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-h2cs6" podUID="d9ea5934-356a-47e5-9fc6-1ae4f39f4e19" Jul 09 15:31:45.967850 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:31:45.967813 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vxr6x" event={"ID":"38e3cf1c-0139-47f3-a0f2-708c4e43b855","Type":"ContainerStarted","Data":"e1dd0e0b52e1f3b8d04a99cd0557647af284bc88eae63f23185a2b18fe56f091"} Jul 09 15:31:45.989600 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:31:45.989533 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vxr6x" podStartSLOduration=3.972211624 podStartE2EDuration="27.989511731s" totalImagesPullingTime="24.017300107s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-09 15:31:18 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 15:31:45.989279582 +0000 UTC m=+27.710786463" watchObservedRunningTime="2026-07-09 15:31:45.989511731 +0000 UTC m=+27.711018603" Jul 09 15:31:46.804987 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:31:46.804959 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-8f2p4" Jul 09 15:31:46.805102 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:31:46.804959 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-sdbbl" Jul 09 15:31:46.805143 ip-10-0-133-183 kubenswrapper[2573]: E0709 15:31:46.805094 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-8f2p4" podUID="8e56f4e6-cbec-422f-9505-c7c6654159d9" Jul 09 15:31:46.805143 ip-10-0-133-183 kubenswrapper[2573]: E0709 15:31:46.805120 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-sdbbl" podUID="e8de13cc-b0e0-4a54-bd44-3ba73cbf2bf2" Jul 09 15:31:46.976225 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:31:46.976194 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-b7b9v_db22f88d-a953-488e-b384-67f991351f73/ovn-acl-logging/0.log" Jul 09 15:31:46.976824 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:31:46.976605 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-b7b9v" event={"ID":"db22f88d-a953-488e-b384-67f991351f73","Type":"ContainerStarted","Data":"6364eae7463fdace05991ba2c631e19c9510bba1a5fd2354c800dda71aba7f2f"} Jul 09 15:31:46.978518 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:31:46.978492 2573 generic.go:356] "Generic (PLEG): container finished" podID="85842275-e9c5-44af-8e99-1700a37602e3" containerID="9f152f066718f80bd35c2379d99790fd92af9b5c881c392a49b61cae614c8c5a" exitCode=0 Jul 09 15:31:46.978628 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:31:46.978574 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-g5qvc" event={"ID":"85842275-e9c5-44af-8e99-1700a37602e3","Type":"ContainerDied","Data":"9f152f066718f80bd35c2379d99790fd92af9b5c881c392a49b61cae614c8c5a"} Jul 09 15:31:47.805528 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:31:47.805498 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-h2cs6" Jul 09 15:31:47.805685 ip-10-0-133-183 kubenswrapper[2573]: E0709 15:31:47.805605 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-h2cs6" podUID="d9ea5934-356a-47e5-9fc6-1ae4f39f4e19" Jul 09 15:31:48.806375 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:31:48.806107 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-8f2p4" Jul 09 15:31:48.806375 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:31:48.806164 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-sdbbl" Jul 09 15:31:48.806375 ip-10-0-133-183 kubenswrapper[2573]: E0709 15:31:48.806358 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-8f2p4" podUID="8e56f4e6-cbec-422f-9505-c7c6654159d9" Jul 09 15:31:48.806988 ip-10-0-133-183 kubenswrapper[2573]: E0709 15:31:48.806454 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-sdbbl" podUID="e8de13cc-b0e0-4a54-bd44-3ba73cbf2bf2" Jul 09 15:31:48.986952 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:31:48.986928 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-b7b9v_db22f88d-a953-488e-b384-67f991351f73/ovn-acl-logging/0.log" Jul 09 15:31:48.987350 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:31:48.987323 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-b7b9v" event={"ID":"db22f88d-a953-488e-b384-67f991351f73","Type":"ContainerStarted","Data":"0c61d5907b01a3470cfd151edf810ead38ba81ddbc4f72fb32c2142b1595c64d"} Jul 09 15:31:48.987557 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:31:48.987541 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-b7b9v" Jul 09 15:31:48.987787 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:31:48.987760 2573 scope.go:117] "RemoveContainer" containerID="19eeface74077b327e95767180802bf9a2fdb5593bada3e61556a95703c87f8d" Jul 09 15:31:48.989970 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:31:48.989906 2573 generic.go:356] "Generic (PLEG): container finished" podID="85842275-e9c5-44af-8e99-1700a37602e3" containerID="c04abaef1804fcbbaec3285dac4f10bdc76fea77f4d6b91c0aaa11968fd3f281" exitCode=0 Jul 09 15:31:48.989970 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:31:48.989942 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-g5qvc" event={"ID":"85842275-e9c5-44af-8e99-1700a37602e3","Type":"ContainerDied","Data":"c04abaef1804fcbbaec3285dac4f10bdc76fea77f4d6b91c0aaa11968fd3f281"} Jul 09 15:31:49.007349 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:31:49.007323 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-b7b9v" Jul 09 15:31:49.810473 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:31:49.806487 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-h2cs6" Jul 09 15:31:49.810473 ip-10-0-133-183 kubenswrapper[2573]: E0709 15:31:49.806649 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-h2cs6" podUID="d9ea5934-356a-47e5-9fc6-1ae4f39f4e19" Jul 09 15:31:49.996297 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:31:49.996266 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-b7b9v_db22f88d-a953-488e-b384-67f991351f73/ovn-acl-logging/0.log" Jul 09 15:31:49.996663 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:31:49.996637 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-b7b9v" event={"ID":"db22f88d-a953-488e-b384-67f991351f73","Type":"ContainerStarted","Data":"1c2437e892c6ee6e135c167fb63721e17422061a294273646e6fdc0d3e75d8a0"} Jul 09 15:31:49.996867 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:31:49.996850 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-b7b9v" Jul 09 15:31:49.996960 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:31:49.996939 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-b7b9v" Jul 09 15:31:50.012048 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:31:50.012025 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-b7b9v" Jul 09 15:31:50.025483 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:31:50.025439 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-b7b9v" podStartSLOduration=9.388612728 podStartE2EDuration="32.025424828s" totalImagesPullingTime="22.6368121s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-09 15:31:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 15:31:50.023942098 +0000 UTC m=+31.745448970" watchObservedRunningTime="2026-07-09 15:31:50.025424828 +0000 UTC m=+31.746931697" Jul 09 15:31:50.572166 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:31:50.572068 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-h2cs6"] Jul 09 15:31:50.572348 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:31:50.572332 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-h2cs6" Jul 09 15:31:50.572553 ip-10-0-133-183 kubenswrapper[2573]: E0709 15:31:50.572528 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-h2cs6" podUID="d9ea5934-356a-47e5-9fc6-1ae4f39f4e19" Jul 09 15:31:50.573156 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:31:50.572863 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-sdbbl"] Jul 09 15:31:50.573156 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:31:50.572991 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-sdbbl" Jul 09 15:31:50.573156 ip-10-0-133-183 kubenswrapper[2573]: E0709 15:31:50.573075 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-sdbbl" podUID="e8de13cc-b0e0-4a54-bd44-3ba73cbf2bf2" Jul 09 15:31:50.573856 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:31:50.573826 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-8f2p4"] Jul 09 15:31:50.574022 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:31:50.573978 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-8f2p4" Jul 09 15:31:50.574129 ip-10-0-133-183 kubenswrapper[2573]: E0709 15:31:50.574104 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-8f2p4" podUID="8e56f4e6-cbec-422f-9505-c7c6654159d9" Jul 09 15:31:52.494181 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:31:52.494140 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/d9ea5934-356a-47e5-9fc6-1ae4f39f4e19-original-pull-secret\") pod \"global-pull-secret-syncer-h2cs6\" (UID: \"d9ea5934-356a-47e5-9fc6-1ae4f39f4e19\") " pod="kube-system/global-pull-secret-syncer-h2cs6" Jul 09 15:31:52.494635 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:31:52.494199 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8e56f4e6-cbec-422f-9505-c7c6654159d9-metrics-certs\") pod \"network-metrics-daemon-8f2p4\" (UID: \"8e56f4e6-cbec-422f-9505-c7c6654159d9\") " pod="openshift-multus/network-metrics-daemon-8f2p4" Jul 09 15:31:52.494635 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:31:52.494234 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l6q9n\" (UniqueName: \"kubernetes.io/projected/e8de13cc-b0e0-4a54-bd44-3ba73cbf2bf2-kube-api-access-l6q9n\") pod \"network-check-target-sdbbl\" (UID: \"e8de13cc-b0e0-4a54-bd44-3ba73cbf2bf2\") " pod="openshift-network-diagnostics/network-check-target-sdbbl" Jul 09 15:31:52.494635 ip-10-0-133-183 kubenswrapper[2573]: E0709 15:31:52.494314 2573 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 09 15:31:52.494635 ip-10-0-133-183 kubenswrapper[2573]: E0709 15:31:52.494362 2573 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 09 15:31:52.494635 ip-10-0-133-183 kubenswrapper[2573]: E0709 15:31:52.494393 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d9ea5934-356a-47e5-9fc6-1ae4f39f4e19-original-pull-secret podName:d9ea5934-356a-47e5-9fc6-1ae4f39f4e19 nodeName:}" failed. No retries permitted until 2026-07-09 15:32:24.494367462 +0000 UTC m=+66.215874310 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/d9ea5934-356a-47e5-9fc6-1ae4f39f4e19-original-pull-secret") pod "global-pull-secret-syncer-h2cs6" (UID: "d9ea5934-356a-47e5-9fc6-1ae4f39f4e19") : object "kube-system"/"original-pull-secret" not registered Jul 09 15:31:52.494635 ip-10-0-133-183 kubenswrapper[2573]: E0709 15:31:52.494431 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8e56f4e6-cbec-422f-9505-c7c6654159d9-metrics-certs podName:8e56f4e6-cbec-422f-9505-c7c6654159d9 nodeName:}" failed. No retries permitted until 2026-07-09 15:32:24.49441234 +0000 UTC m=+66.215919191 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/8e56f4e6-cbec-422f-9505-c7c6654159d9-metrics-certs") pod "network-metrics-daemon-8f2p4" (UID: "8e56f4e6-cbec-422f-9505-c7c6654159d9") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 09 15:31:52.494635 ip-10-0-133-183 kubenswrapper[2573]: E0709 15:31:52.494364 2573 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 09 15:31:52.494635 ip-10-0-133-183 kubenswrapper[2573]: E0709 15:31:52.494459 2573 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 09 15:31:52.494635 ip-10-0-133-183 kubenswrapper[2573]: E0709 15:31:52.494474 2573 projected.go:196] Error preparing data for projected volume kube-api-access-l6q9n for pod openshift-network-diagnostics/network-check-target-sdbbl: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 09 15:31:52.494635 ip-10-0-133-183 kubenswrapper[2573]: E0709 15:31:52.494511 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/e8de13cc-b0e0-4a54-bd44-3ba73cbf2bf2-kube-api-access-l6q9n podName:e8de13cc-b0e0-4a54-bd44-3ba73cbf2bf2 nodeName:}" failed. No retries permitted until 2026-07-09 15:32:24.494499238 +0000 UTC m=+66.216006089 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-l6q9n" (UniqueName: "kubernetes.io/projected/e8de13cc-b0e0-4a54-bd44-3ba73cbf2bf2-kube-api-access-l6q9n") pod "network-check-target-sdbbl" (UID: "e8de13cc-b0e0-4a54-bd44-3ba73cbf2bf2") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 09 15:31:52.805564 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:31:52.805476 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-h2cs6" Jul 09 15:31:52.805716 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:31:52.805477 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-sdbbl" Jul 09 15:31:52.805716 ip-10-0-133-183 kubenswrapper[2573]: E0709 15:31:52.805602 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-h2cs6" podUID="d9ea5934-356a-47e5-9fc6-1ae4f39f4e19" Jul 09 15:31:52.805716 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:31:52.805487 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-8f2p4" Jul 09 15:31:52.805716 ip-10-0-133-183 kubenswrapper[2573]: E0709 15:31:52.805660 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-sdbbl" podUID="e8de13cc-b0e0-4a54-bd44-3ba73cbf2bf2" Jul 09 15:31:52.805937 ip-10-0-133-183 kubenswrapper[2573]: E0709 15:31:52.805767 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-8f2p4" podUID="8e56f4e6-cbec-422f-9505-c7c6654159d9" Jul 09 15:31:54.805754 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:31:54.805719 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-h2cs6" Jul 09 15:31:54.806127 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:31:54.805719 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-sdbbl" Jul 09 15:31:54.806127 ip-10-0-133-183 kubenswrapper[2573]: E0709 15:31:54.805829 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-h2cs6" podUID="d9ea5934-356a-47e5-9fc6-1ae4f39f4e19" Jul 09 15:31:54.806127 ip-10-0-133-183 kubenswrapper[2573]: E0709 15:31:54.805933 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-sdbbl" podUID="e8de13cc-b0e0-4a54-bd44-3ba73cbf2bf2" Jul 09 15:31:54.806127 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:31:54.805725 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-8f2p4" Jul 09 15:31:54.806127 ip-10-0-133-183 kubenswrapper[2573]: E0709 15:31:54.806074 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-8f2p4" podUID="8e56f4e6-cbec-422f-9505-c7c6654159d9" Jul 09 15:31:56.013288 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:31:56.013252 2573 generic.go:356] "Generic (PLEG): container finished" podID="85842275-e9c5-44af-8e99-1700a37602e3" containerID="d469770a794648e72ee90e37b2c3c2cee275e7dcbbd760e878e2bbb853df588c" exitCode=0 Jul 09 15:31:56.013806 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:31:56.013331 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-g5qvc" event={"ID":"85842275-e9c5-44af-8e99-1700a37602e3","Type":"ContainerDied","Data":"d469770a794648e72ee90e37b2c3c2cee275e7dcbbd760e878e2bbb853df588c"} Jul 09 15:31:56.626927 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:31:56.626900 2573 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-183.ec2.internal" event="NodeReady" Jul 09 15:31:56.627127 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:31:56.627037 2573 kubelet_node_status.go:489] "Fast updating node status as it just became ready" Jul 09 15:31:56.666237 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:31:56.666200 2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-7c768548b6-5tl98"] Jul 09 15:31:56.676127 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:31:56.676104 2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-ffqv6"] Jul 09 15:31:56.676250 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:31:56.676227 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-7c768548b6-5tl98" Jul 09 15:31:56.678979 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:31:56.678958 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\"" Jul 09 15:31:56.679512 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:31:56.679483 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-lq2hc\"" Jul 09 15:31:56.679661 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:31:56.679618 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\"" Jul 09 15:31:56.679661 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:31:56.679629 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\"" Jul 09 15:31:56.690061 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:31:56.688573 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-7c768548b6-5tl98"] Jul 09 15:31:56.690061 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:31:56.688634 2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-fnfp8"] Jul 09 15:31:56.690061 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:31:56.689846 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-ffqv6" Jul 09 15:31:56.690495 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:31:56.690472 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\"" Jul 09 15:31:56.693178 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:31:56.693153 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\"" Jul 09 15:31:56.693282 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:31:56.693161 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-zcxss\"" Jul 09 15:31:56.693411 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:31:56.693393 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\"" Jul 09 15:31:56.699230 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:31:56.699208 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-ffqv6"] Jul 09 15:31:56.699351 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:31:56.699241 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-fnfp8"] Jul 09 15:31:56.699351 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:31:56.699312 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-fnfp8" Jul 09 15:31:56.702573 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:31:56.702552 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\"" Jul 09 15:31:56.702697 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:31:56.702590 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\"" Jul 09 15:31:56.702697 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:31:56.702599 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\"" Jul 09 15:31:56.702788 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:31:56.702706 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-wrjlr\"" Jul 09 15:31:56.805690 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:31:56.805656 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-sdbbl" Jul 09 15:31:56.805909 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:31:56.805694 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-8f2p4" Jul 09 15:31:56.805909 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:31:56.805659 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-h2cs6" Jul 09 15:31:56.809011 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:31:56.808972 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Jul 09 15:31:56.809188 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:31:56.809024 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Jul 09 15:31:56.809188 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:31:56.809034 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Jul 09 15:31:56.809188 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:31:56.809043 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-bnkz7\"" Jul 09 15:31:56.809188 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:31:56.809117 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-rsq6f\"" Jul 09 15:31:56.809188 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:31:56.809163 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Jul 09 15:31:56.825648 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:31:56.825628 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/d4b4e91f-6d12-45a0-a779-3fcda0e75f08-cert\") pod \"ingress-canary-fnfp8\" (UID: \"d4b4e91f-6d12-45a0-a779-3fcda0e75f08\") " pod="openshift-ingress-canary/ingress-canary-fnfp8" Jul 09 15:31:56.825762 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:31:56.825683 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/81be5afd-69ba-4d11-95d0-efdb0f62bad0-registry-certificates\") pod \"image-registry-7c768548b6-5tl98\" (UID: \"81be5afd-69ba-4d11-95d0-efdb0f62bad0\") " pod="openshift-image-registry/image-registry-7c768548b6-5tl98" Jul 09 15:31:56.825762 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:31:56.825711 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pzd2p\" (UniqueName: \"kubernetes.io/projected/b9e43e24-97dc-464d-a445-2100e1b546a9-kube-api-access-pzd2p\") pod \"dns-default-ffqv6\" (UID: \"b9e43e24-97dc-464d-a445-2100e1b546a9\") " pod="openshift-dns/dns-default-ffqv6" Jul 09 15:31:56.825869 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:31:56.825749 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/81be5afd-69ba-4d11-95d0-efdb0f62bad0-image-registry-private-configuration\") pod \"image-registry-7c768548b6-5tl98\" (UID: \"81be5afd-69ba-4d11-95d0-efdb0f62bad0\") " pod="openshift-image-registry/image-registry-7c768548b6-5tl98" Jul 09 15:31:56.825869 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:31:56.825796 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/81be5afd-69ba-4d11-95d0-efdb0f62bad0-bound-sa-token\") pod \"image-registry-7c768548b6-5tl98\" (UID: \"81be5afd-69ba-4d11-95d0-efdb0f62bad0\") " pod="openshift-image-registry/image-registry-7c768548b6-5tl98" Jul 09 15:31:56.825869 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:31:56.825820 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q8mdg\" (UniqueName: \"kubernetes.io/projected/81be5afd-69ba-4d11-95d0-efdb0f62bad0-kube-api-access-q8mdg\") pod \"image-registry-7c768548b6-5tl98\" (UID: \"81be5afd-69ba-4d11-95d0-efdb0f62bad0\") " pod="openshift-image-registry/image-registry-7c768548b6-5tl98" Jul 09 15:31:56.825869 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:31:56.825842 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/b9e43e24-97dc-464d-a445-2100e1b546a9-config-volume\") pod \"dns-default-ffqv6\" (UID: \"b9e43e24-97dc-464d-a445-2100e1b546a9\") " pod="openshift-dns/dns-default-ffqv6" Jul 09 15:31:56.826045 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:31:56.825896 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/81be5afd-69ba-4d11-95d0-efdb0f62bad0-registry-tls\") pod \"image-registry-7c768548b6-5tl98\" (UID: \"81be5afd-69ba-4d11-95d0-efdb0f62bad0\") " pod="openshift-image-registry/image-registry-7c768548b6-5tl98" Jul 09 15:31:56.826045 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:31:56.825917 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b9e43e24-97dc-464d-a445-2100e1b546a9-tmp-dir\") pod \"dns-default-ffqv6\" (UID: \"b9e43e24-97dc-464d-a445-2100e1b546a9\") " pod="openshift-dns/dns-default-ffqv6" Jul 09 15:31:56.826045 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:31:56.825962 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/81be5afd-69ba-4d11-95d0-efdb0f62bad0-ca-trust-extracted\") pod \"image-registry-7c768548b6-5tl98\" (UID: \"81be5afd-69ba-4d11-95d0-efdb0f62bad0\") " pod="openshift-image-registry/image-registry-7c768548b6-5tl98" Jul 09 15:31:56.826045 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:31:56.825980 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/81be5afd-69ba-4d11-95d0-efdb0f62bad0-trusted-ca\") pod \"image-registry-7c768548b6-5tl98\" (UID: \"81be5afd-69ba-4d11-95d0-efdb0f62bad0\") " pod="openshift-image-registry/image-registry-7c768548b6-5tl98" Jul 09 15:31:56.826045 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:31:56.825995 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bv5vw\" (UniqueName: \"kubernetes.io/projected/d4b4e91f-6d12-45a0-a779-3fcda0e75f08-kube-api-access-bv5vw\") pod \"ingress-canary-fnfp8\" (UID: \"d4b4e91f-6d12-45a0-a779-3fcda0e75f08\") " pod="openshift-ingress-canary/ingress-canary-fnfp8" Jul 09 15:31:56.826045 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:31:56.826011 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/81be5afd-69ba-4d11-95d0-efdb0f62bad0-installation-pull-secrets\") pod \"image-registry-7c768548b6-5tl98\" (UID: \"81be5afd-69ba-4d11-95d0-efdb0f62bad0\") " pod="openshift-image-registry/image-registry-7c768548b6-5tl98" Jul 09 15:31:56.826045 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:31:56.826028 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/b9e43e24-97dc-464d-a445-2100e1b546a9-metrics-tls\") pod \"dns-default-ffqv6\" (UID: \"b9e43e24-97dc-464d-a445-2100e1b546a9\") " pod="openshift-dns/dns-default-ffqv6" Jul 09 15:31:56.927176 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:31:56.927092 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/81be5afd-69ba-4d11-95d0-efdb0f62bad0-ca-trust-extracted\") pod \"image-registry-7c768548b6-5tl98\" (UID: \"81be5afd-69ba-4d11-95d0-efdb0f62bad0\") " pod="openshift-image-registry/image-registry-7c768548b6-5tl98" Jul 09 15:31:56.927176 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:31:56.927135 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/81be5afd-69ba-4d11-95d0-efdb0f62bad0-trusted-ca\") pod \"image-registry-7c768548b6-5tl98\" (UID: \"81be5afd-69ba-4d11-95d0-efdb0f62bad0\") " pod="openshift-image-registry/image-registry-7c768548b6-5tl98" Jul 09 15:31:56.927176 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:31:56.927163 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bv5vw\" (UniqueName: \"kubernetes.io/projected/d4b4e91f-6d12-45a0-a779-3fcda0e75f08-kube-api-access-bv5vw\") pod \"ingress-canary-fnfp8\" (UID: \"d4b4e91f-6d12-45a0-a779-3fcda0e75f08\") " pod="openshift-ingress-canary/ingress-canary-fnfp8" Jul 09 15:31:56.927503 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:31:56.927192 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/81be5afd-69ba-4d11-95d0-efdb0f62bad0-installation-pull-secrets\") pod \"image-registry-7c768548b6-5tl98\" (UID: \"81be5afd-69ba-4d11-95d0-efdb0f62bad0\") " pod="openshift-image-registry/image-registry-7c768548b6-5tl98" Jul 09 15:31:56.927503 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:31:56.927216 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/b9e43e24-97dc-464d-a445-2100e1b546a9-metrics-tls\") pod \"dns-default-ffqv6\" (UID: \"b9e43e24-97dc-464d-a445-2100e1b546a9\") " pod="openshift-dns/dns-default-ffqv6" Jul 09 15:31:56.927503 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:31:56.927268 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/d4b4e91f-6d12-45a0-a779-3fcda0e75f08-cert\") pod \"ingress-canary-fnfp8\" (UID: \"d4b4e91f-6d12-45a0-a779-3fcda0e75f08\") " pod="openshift-ingress-canary/ingress-canary-fnfp8" Jul 09 15:31:56.927503 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:31:56.927321 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/81be5afd-69ba-4d11-95d0-efdb0f62bad0-registry-certificates\") pod \"image-registry-7c768548b6-5tl98\" (UID: \"81be5afd-69ba-4d11-95d0-efdb0f62bad0\") " pod="openshift-image-registry/image-registry-7c768548b6-5tl98" Jul 09 15:31:56.927503 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:31:56.927346 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pzd2p\" (UniqueName: \"kubernetes.io/projected/b9e43e24-97dc-464d-a445-2100e1b546a9-kube-api-access-pzd2p\") pod \"dns-default-ffqv6\" (UID: \"b9e43e24-97dc-464d-a445-2100e1b546a9\") " pod="openshift-dns/dns-default-ffqv6" Jul 09 15:31:56.927503 ip-10-0-133-183 kubenswrapper[2573]: E0709 15:31:56.927370 2573 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 09 15:31:56.927503 ip-10-0-133-183 kubenswrapper[2573]: E0709 15:31:56.927468 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b9e43e24-97dc-464d-a445-2100e1b546a9-metrics-tls podName:b9e43e24-97dc-464d-a445-2100e1b546a9 nodeName:}" failed. No retries permitted until 2026-07-09 15:31:57.427444123 +0000 UTC m=+39.148950990 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/b9e43e24-97dc-464d-a445-2100e1b546a9-metrics-tls") pod "dns-default-ffqv6" (UID: "b9e43e24-97dc-464d-a445-2100e1b546a9") : secret "dns-default-metrics-tls" not found Jul 09 15:31:56.927845 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:31:56.927375 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/81be5afd-69ba-4d11-95d0-efdb0f62bad0-image-registry-private-configuration\") pod \"image-registry-7c768548b6-5tl98\" (UID: \"81be5afd-69ba-4d11-95d0-efdb0f62bad0\") " pod="openshift-image-registry/image-registry-7c768548b6-5tl98" Jul 09 15:31:56.927845 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:31:56.927548 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/81be5afd-69ba-4d11-95d0-efdb0f62bad0-bound-sa-token\") pod \"image-registry-7c768548b6-5tl98\" (UID: \"81be5afd-69ba-4d11-95d0-efdb0f62bad0\") " pod="openshift-image-registry/image-registry-7c768548b6-5tl98" Jul 09 15:31:56.927845 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:31:56.927571 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/81be5afd-69ba-4d11-95d0-efdb0f62bad0-ca-trust-extracted\") pod \"image-registry-7c768548b6-5tl98\" (UID: \"81be5afd-69ba-4d11-95d0-efdb0f62bad0\") " pod="openshift-image-registry/image-registry-7c768548b6-5tl98" Jul 09 15:31:56.927845 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:31:56.927576 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q8mdg\" (UniqueName: \"kubernetes.io/projected/81be5afd-69ba-4d11-95d0-efdb0f62bad0-kube-api-access-q8mdg\") pod \"image-registry-7c768548b6-5tl98\" (UID: \"81be5afd-69ba-4d11-95d0-efdb0f62bad0\") " pod="openshift-image-registry/image-registry-7c768548b6-5tl98" Jul 09 15:31:56.927845 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:31:56.927630 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/b9e43e24-97dc-464d-a445-2100e1b546a9-config-volume\") pod \"dns-default-ffqv6\" (UID: \"b9e43e24-97dc-464d-a445-2100e1b546a9\") " pod="openshift-dns/dns-default-ffqv6" Jul 09 15:31:56.927845 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:31:56.927662 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/81be5afd-69ba-4d11-95d0-efdb0f62bad0-registry-tls\") pod \"image-registry-7c768548b6-5tl98\" (UID: \"81be5afd-69ba-4d11-95d0-efdb0f62bad0\") " pod="openshift-image-registry/image-registry-7c768548b6-5tl98" Jul 09 15:31:56.927845 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:31:56.927687 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b9e43e24-97dc-464d-a445-2100e1b546a9-tmp-dir\") pod \"dns-default-ffqv6\" (UID: \"b9e43e24-97dc-464d-a445-2100e1b546a9\") " pod="openshift-dns/dns-default-ffqv6" Jul 09 15:31:56.928197 ip-10-0-133-183 kubenswrapper[2573]: E0709 15:31:56.927971 2573 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 09 15:31:56.928197 ip-10-0-133-183 kubenswrapper[2573]: E0709 15:31:56.927987 2573 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-7c768548b6-5tl98: secret "image-registry-tls" not found Jul 09 15:31:56.928197 ip-10-0-133-183 kubenswrapper[2573]: E0709 15:31:56.928072 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/81be5afd-69ba-4d11-95d0-efdb0f62bad0-registry-tls podName:81be5afd-69ba-4d11-95d0-efdb0f62bad0 nodeName:}" failed. No retries permitted until 2026-07-09 15:31:57.428052311 +0000 UTC m=+39.149559167 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/81be5afd-69ba-4d11-95d0-efdb0f62bad0-registry-tls") pod "image-registry-7c768548b6-5tl98" (UID: "81be5afd-69ba-4d11-95d0-efdb0f62bad0") : secret "image-registry-tls" not found Jul 09 15:31:56.928197 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:31:56.928075 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b9e43e24-97dc-464d-a445-2100e1b546a9-tmp-dir\") pod \"dns-default-ffqv6\" (UID: \"b9e43e24-97dc-464d-a445-2100e1b546a9\") " pod="openshift-dns/dns-default-ffqv6" Jul 09 15:31:56.928427 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:31:56.928270 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/b9e43e24-97dc-464d-a445-2100e1b546a9-config-volume\") pod \"dns-default-ffqv6\" (UID: \"b9e43e24-97dc-464d-a445-2100e1b546a9\") " pod="openshift-dns/dns-default-ffqv6" Jul 09 15:31:56.928490 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:31:56.928472 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/81be5afd-69ba-4d11-95d0-efdb0f62bad0-registry-certificates\") pod \"image-registry-7c768548b6-5tl98\" (UID: \"81be5afd-69ba-4d11-95d0-efdb0f62bad0\") " pod="openshift-image-registry/image-registry-7c768548b6-5tl98" Jul 09 15:31:56.928616 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:31:56.928597 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/81be5afd-69ba-4d11-95d0-efdb0f62bad0-trusted-ca\") pod \"image-registry-7c768548b6-5tl98\" (UID: \"81be5afd-69ba-4d11-95d0-efdb0f62bad0\") " pod="openshift-image-registry/image-registry-7c768548b6-5tl98" Jul 09 15:31:56.931571 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:31:56.931547 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/81be5afd-69ba-4d11-95d0-efdb0f62bad0-image-registry-private-configuration\") pod \"image-registry-7c768548b6-5tl98\" (UID: \"81be5afd-69ba-4d11-95d0-efdb0f62bad0\") " pod="openshift-image-registry/image-registry-7c768548b6-5tl98" Jul 09 15:31:56.931676 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:31:56.931595 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/81be5afd-69ba-4d11-95d0-efdb0f62bad0-installation-pull-secrets\") pod \"image-registry-7c768548b6-5tl98\" (UID: \"81be5afd-69ba-4d11-95d0-efdb0f62bad0\") " pod="openshift-image-registry/image-registry-7c768548b6-5tl98" Jul 09 15:31:56.931714 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:31:56.931680 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/d4b4e91f-6d12-45a0-a779-3fcda0e75f08-cert\") pod \"ingress-canary-fnfp8\" (UID: \"d4b4e91f-6d12-45a0-a779-3fcda0e75f08\") " pod="openshift-ingress-canary/ingress-canary-fnfp8" Jul 09 15:31:56.948452 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:31:56.948416 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pzd2p\" (UniqueName: \"kubernetes.io/projected/b9e43e24-97dc-464d-a445-2100e1b546a9-kube-api-access-pzd2p\") pod \"dns-default-ffqv6\" (UID: \"b9e43e24-97dc-464d-a445-2100e1b546a9\") " pod="openshift-dns/dns-default-ffqv6" Jul 09 15:31:56.949078 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:31:56.949049 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bv5vw\" (UniqueName: \"kubernetes.io/projected/d4b4e91f-6d12-45a0-a779-3fcda0e75f08-kube-api-access-bv5vw\") pod \"ingress-canary-fnfp8\" (UID: \"d4b4e91f-6d12-45a0-a779-3fcda0e75f08\") " pod="openshift-ingress-canary/ingress-canary-fnfp8" Jul 09 15:31:56.949175 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:31:56.949115 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-q8mdg\" (UniqueName: \"kubernetes.io/projected/81be5afd-69ba-4d11-95d0-efdb0f62bad0-kube-api-access-q8mdg\") pod \"image-registry-7c768548b6-5tl98\" (UID: \"81be5afd-69ba-4d11-95d0-efdb0f62bad0\") " pod="openshift-image-registry/image-registry-7c768548b6-5tl98" Jul 09 15:31:56.949376 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:31:56.949361 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/81be5afd-69ba-4d11-95d0-efdb0f62bad0-bound-sa-token\") pod \"image-registry-7c768548b6-5tl98\" (UID: \"81be5afd-69ba-4d11-95d0-efdb0f62bad0\") " pod="openshift-image-registry/image-registry-7c768548b6-5tl98" Jul 09 15:31:57.008698 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:31:57.008666 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-fnfp8" Jul 09 15:31:57.018658 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:31:57.018627 2573 generic.go:356] "Generic (PLEG): container finished" podID="85842275-e9c5-44af-8e99-1700a37602e3" containerID="7ddff23cc3f3ce24598a7ad31f700f109b8da8f1cf517bb8739d4be83acc08ba" exitCode=0 Jul 09 15:31:57.019166 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:31:57.018675 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-g5qvc" event={"ID":"85842275-e9c5-44af-8e99-1700a37602e3","Type":"ContainerDied","Data":"7ddff23cc3f3ce24598a7ad31f700f109b8da8f1cf517bb8739d4be83acc08ba"} Jul 09 15:31:57.202084 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:31:57.201916 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-fnfp8"] Jul 09 15:31:57.205981 ip-10-0-133-183 kubenswrapper[2573]: W0709 15:31:57.205952 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd4b4e91f_6d12_45a0_a779_3fcda0e75f08.slice/crio-47f4efdeec34b3f4b81460d2ef2bd503a96cb726929d54c40b462ef823d3c5fa WatchSource:0}: Error finding container 47f4efdeec34b3f4b81460d2ef2bd503a96cb726929d54c40b462ef823d3c5fa: Status 404 returned error can't find the container with id 47f4efdeec34b3f4b81460d2ef2bd503a96cb726929d54c40b462ef823d3c5fa Jul 09 15:31:57.432593 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:31:57.432554 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/81be5afd-69ba-4d11-95d0-efdb0f62bad0-registry-tls\") pod \"image-registry-7c768548b6-5tl98\" (UID: \"81be5afd-69ba-4d11-95d0-efdb0f62bad0\") " pod="openshift-image-registry/image-registry-7c768548b6-5tl98" Jul 09 15:31:57.432762 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:31:57.432622 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/b9e43e24-97dc-464d-a445-2100e1b546a9-metrics-tls\") pod \"dns-default-ffqv6\" (UID: \"b9e43e24-97dc-464d-a445-2100e1b546a9\") " pod="openshift-dns/dns-default-ffqv6" Jul 09 15:31:57.432762 ip-10-0-133-183 kubenswrapper[2573]: E0709 15:31:57.432713 2573 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 09 15:31:57.432762 ip-10-0-133-183 kubenswrapper[2573]: E0709 15:31:57.432734 2573 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-7c768548b6-5tl98: secret "image-registry-tls" not found Jul 09 15:31:57.432762 ip-10-0-133-183 kubenswrapper[2573]: E0709 15:31:57.432714 2573 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 09 15:31:57.432905 ip-10-0-133-183 kubenswrapper[2573]: E0709 15:31:57.432797 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/81be5afd-69ba-4d11-95d0-efdb0f62bad0-registry-tls podName:81be5afd-69ba-4d11-95d0-efdb0f62bad0 nodeName:}" failed. No retries permitted until 2026-07-09 15:31:58.432782021 +0000 UTC m=+40.154288872 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/81be5afd-69ba-4d11-95d0-efdb0f62bad0-registry-tls") pod "image-registry-7c768548b6-5tl98" (UID: "81be5afd-69ba-4d11-95d0-efdb0f62bad0") : secret "image-registry-tls" not found Jul 09 15:31:57.432905 ip-10-0-133-183 kubenswrapper[2573]: E0709 15:31:57.432812 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b9e43e24-97dc-464d-a445-2100e1b546a9-metrics-tls podName:b9e43e24-97dc-464d-a445-2100e1b546a9 nodeName:}" failed. No retries permitted until 2026-07-09 15:31:58.43280648 +0000 UTC m=+40.154313326 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/b9e43e24-97dc-464d-a445-2100e1b546a9-metrics-tls") pod "dns-default-ffqv6" (UID: "b9e43e24-97dc-464d-a445-2100e1b546a9") : secret "dns-default-metrics-tls" not found Jul 09 15:31:58.025863 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:31:58.025820 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-g5qvc" event={"ID":"85842275-e9c5-44af-8e99-1700a37602e3","Type":"ContainerStarted","Data":"308a5d616f21b37b39de4f7665bcccf00db874c9570583372bf9e8896fa95bae"} Jul 09 15:31:58.027070 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:31:58.027038 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-fnfp8" event={"ID":"d4b4e91f-6d12-45a0-a779-3fcda0e75f08","Type":"ContainerStarted","Data":"47f4efdeec34b3f4b81460d2ef2bd503a96cb726929d54c40b462ef823d3c5fa"} Jul 09 15:31:58.053796 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:31:58.053736 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-g5qvc" podStartSLOduration=9.402329 podStartE2EDuration="40.053717608s" totalImagesPullingTime="30.651388608s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-09 15:31:18 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 15:31:58.05142523 +0000 UTC m=+39.772932103" watchObservedRunningTime="2026-07-09 15:31:58.053717608 +0000 UTC m=+39.775224478" Jul 09 15:31:58.445242 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:31:58.445209 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/b9e43e24-97dc-464d-a445-2100e1b546a9-metrics-tls\") pod \"dns-default-ffqv6\" (UID: \"b9e43e24-97dc-464d-a445-2100e1b546a9\") " pod="openshift-dns/dns-default-ffqv6" Jul 09 15:31:58.445420 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:31:58.445285 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/81be5afd-69ba-4d11-95d0-efdb0f62bad0-registry-tls\") pod \"image-registry-7c768548b6-5tl98\" (UID: \"81be5afd-69ba-4d11-95d0-efdb0f62bad0\") " pod="openshift-image-registry/image-registry-7c768548b6-5tl98" Jul 09 15:31:58.445420 ip-10-0-133-183 kubenswrapper[2573]: E0709 15:31:58.445364 2573 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 09 15:31:58.445420 ip-10-0-133-183 kubenswrapper[2573]: E0709 15:31:58.445404 2573 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 09 15:31:58.445581 ip-10-0-133-183 kubenswrapper[2573]: E0709 15:31:58.445427 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b9e43e24-97dc-464d-a445-2100e1b546a9-metrics-tls podName:b9e43e24-97dc-464d-a445-2100e1b546a9 nodeName:}" failed. No retries permitted until 2026-07-09 15:32:00.445406938 +0000 UTC m=+42.166913785 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/b9e43e24-97dc-464d-a445-2100e1b546a9-metrics-tls") pod "dns-default-ffqv6" (UID: "b9e43e24-97dc-464d-a445-2100e1b546a9") : secret "dns-default-metrics-tls" not found Jul 09 15:31:58.445581 ip-10-0-133-183 kubenswrapper[2573]: E0709 15:31:58.445431 2573 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-7c768548b6-5tl98: secret "image-registry-tls" not found Jul 09 15:31:58.445581 ip-10-0-133-183 kubenswrapper[2573]: E0709 15:31:58.445490 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/81be5afd-69ba-4d11-95d0-efdb0f62bad0-registry-tls podName:81be5afd-69ba-4d11-95d0-efdb0f62bad0 nodeName:}" failed. No retries permitted until 2026-07-09 15:32:00.445470001 +0000 UTC m=+42.166976853 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/81be5afd-69ba-4d11-95d0-efdb0f62bad0-registry-tls") pod "image-registry-7c768548b6-5tl98" (UID: "81be5afd-69ba-4d11-95d0-efdb0f62bad0") : secret "image-registry-tls" not found Jul 09 15:31:59.958765 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:31:59.958725 2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-6d4489575b-4x9ks"] Jul 09 15:31:59.966479 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:31:59.966408 2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-7b7744d957-k6dtg"] Jul 09 15:31:59.966627 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:31:59.966528 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-6d4489575b-4x9ks" Jul 09 15:31:59.970802 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:31:59.970773 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"kube-root-ca.crt\"" Jul 09 15:31:59.970802 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:31:59.970799 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"open-cluster-management-image-pull-credentials\"" Jul 09 15:31:59.971019 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:31:59.970799 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"managed-serviceaccount-dockercfg-d6znt\"" Jul 09 15:31:59.971019 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:31:59.970850 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"managed-serviceaccount-hub-kubeconfig\"" Jul 09 15:31:59.971019 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:31:59.970866 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"openshift-service-ca.crt\"" Jul 09 15:31:59.978554 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:31:59.978533 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-6d4489575b-4x9ks"] Jul 09 15:31:59.978696 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:31:59.978560 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-7b7744d957-k6dtg"] Jul 09 15:31:59.978696 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:31:59.978644 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-7b7744d957-k6dtg" Jul 09 15:31:59.981678 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:31:59.981659 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"work-manager-hub-kubeconfig\"" Jul 09 15:32:00.033977 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:32:00.033944 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-fnfp8" event={"ID":"d4b4e91f-6d12-45a0-a779-3fcda0e75f08","Type":"ContainerStarted","Data":"2d4db2ca6b7f203d39a24dedaeddfd01defe8435cc53909751ddf691a8d59b04"} Jul 09 15:32:00.056902 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:32:00.056837 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-fnfp8" podStartSLOduration=1.548658596 podStartE2EDuration="4.056818587s" totalImagesPullingTime="2.508159991s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 15:31:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 15:32:00.056182246 +0000 UTC m=+41.777689134" watchObservedRunningTime="2026-07-09 15:32:00.056818587 +0000 UTC m=+41.778325510" Jul 09 15:32:00.058463 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:32:00.058440 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l6hf6\" (UniqueName: \"kubernetes.io/projected/48292f30-aa88-45e0-8bed-4c675ec638b6-kube-api-access-l6hf6\") pod \"managed-serviceaccount-addon-agent-6d4489575b-4x9ks\" (UID: \"48292f30-aa88-45e0-8bed-4c675ec638b6\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-6d4489575b-4x9ks" Jul 09 15:32:00.058534 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:32:00.058502 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/2c459d95-3f7c-4b06-b0ce-7e938d6c2c48-tmp\") pod \"klusterlet-addon-workmgr-7b7744d957-k6dtg\" (UID: \"2c459d95-3f7c-4b06-b0ce-7e938d6c2c48\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-7b7744d957-k6dtg" Jul 09 15:32:00.058623 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:32:00.058601 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/2c459d95-3f7c-4b06-b0ce-7e938d6c2c48-klusterlet-config\") pod \"klusterlet-addon-workmgr-7b7744d957-k6dtg\" (UID: \"2c459d95-3f7c-4b06-b0ce-7e938d6c2c48\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-7b7744d957-k6dtg" Jul 09 15:32:00.058675 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:32:00.058660 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/48292f30-aa88-45e0-8bed-4c675ec638b6-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-6d4489575b-4x9ks\" (UID: \"48292f30-aa88-45e0-8bed-4c675ec638b6\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-6d4489575b-4x9ks" Jul 09 15:32:00.058727 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:32:00.058712 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h647s\" (UniqueName: \"kubernetes.io/projected/2c459d95-3f7c-4b06-b0ce-7e938d6c2c48-kube-api-access-h647s\") pod \"klusterlet-addon-workmgr-7b7744d957-k6dtg\" (UID: \"2c459d95-3f7c-4b06-b0ce-7e938d6c2c48\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-7b7744d957-k6dtg" Jul 09 15:32:00.159549 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:32:00.159516 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/2c459d95-3f7c-4b06-b0ce-7e938d6c2c48-tmp\") pod \"klusterlet-addon-workmgr-7b7744d957-k6dtg\" (UID: \"2c459d95-3f7c-4b06-b0ce-7e938d6c2c48\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-7b7744d957-k6dtg" Jul 09 15:32:00.159752 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:32:00.159575 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/2c459d95-3f7c-4b06-b0ce-7e938d6c2c48-klusterlet-config\") pod \"klusterlet-addon-workmgr-7b7744d957-k6dtg\" (UID: \"2c459d95-3f7c-4b06-b0ce-7e938d6c2c48\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-7b7744d957-k6dtg" Jul 09 15:32:00.159752 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:32:00.159603 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/48292f30-aa88-45e0-8bed-4c675ec638b6-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-6d4489575b-4x9ks\" (UID: \"48292f30-aa88-45e0-8bed-4c675ec638b6\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-6d4489575b-4x9ks" Jul 09 15:32:00.159872 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:32:00.159758 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h647s\" (UniqueName: \"kubernetes.io/projected/2c459d95-3f7c-4b06-b0ce-7e938d6c2c48-kube-api-access-h647s\") pod \"klusterlet-addon-workmgr-7b7744d957-k6dtg\" (UID: \"2c459d95-3f7c-4b06-b0ce-7e938d6c2c48\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-7b7744d957-k6dtg" Jul 09 15:32:00.159872 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:32:00.159832 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l6hf6\" (UniqueName: \"kubernetes.io/projected/48292f30-aa88-45e0-8bed-4c675ec638b6-kube-api-access-l6hf6\") pod \"managed-serviceaccount-addon-agent-6d4489575b-4x9ks\" (UID: \"48292f30-aa88-45e0-8bed-4c675ec638b6\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-6d4489575b-4x9ks" Jul 09 15:32:00.160061 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:32:00.160039 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/2c459d95-3f7c-4b06-b0ce-7e938d6c2c48-tmp\") pod \"klusterlet-addon-workmgr-7b7744d957-k6dtg\" (UID: \"2c459d95-3f7c-4b06-b0ce-7e938d6c2c48\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-7b7744d957-k6dtg" Jul 09 15:32:00.163601 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:32:00.163569 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/2c459d95-3f7c-4b06-b0ce-7e938d6c2c48-klusterlet-config\") pod \"klusterlet-addon-workmgr-7b7744d957-k6dtg\" (UID: \"2c459d95-3f7c-4b06-b0ce-7e938d6c2c48\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-7b7744d957-k6dtg" Jul 09 15:32:00.163702 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:32:00.163606 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/48292f30-aa88-45e0-8bed-4c675ec638b6-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-6d4489575b-4x9ks\" (UID: \"48292f30-aa88-45e0-8bed-4c675ec638b6\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-6d4489575b-4x9ks" Jul 09 15:32:00.169564 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:32:00.169537 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-l6hf6\" (UniqueName: \"kubernetes.io/projected/48292f30-aa88-45e0-8bed-4c675ec638b6-kube-api-access-l6hf6\") pod \"managed-serviceaccount-addon-agent-6d4489575b-4x9ks\" (UID: \"48292f30-aa88-45e0-8bed-4c675ec638b6\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-6d4489575b-4x9ks" Jul 09 15:32:00.169661 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:32:00.169640 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-h647s\" (UniqueName: \"kubernetes.io/projected/2c459d95-3f7c-4b06-b0ce-7e938d6c2c48-kube-api-access-h647s\") pod \"klusterlet-addon-workmgr-7b7744d957-k6dtg\" (UID: \"2c459d95-3f7c-4b06-b0ce-7e938d6c2c48\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-7b7744d957-k6dtg" Jul 09 15:32:00.294002 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:32:00.293910 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-6d4489575b-4x9ks" Jul 09 15:32:00.302131 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:32:00.302098 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-7b7744d957-k6dtg" Jul 09 15:32:00.442705 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:32:00.442673 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-6d4489575b-4x9ks"] Jul 09 15:32:00.445717 ip-10-0-133-183 kubenswrapper[2573]: W0709 15:32:00.445689 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod48292f30_aa88_45e0_8bed_4c675ec638b6.slice/crio-aaca9b3a3f9d594fd50801c2571595d15f4d9c339cbf774db6480b7626e08dfa WatchSource:0}: Error finding container aaca9b3a3f9d594fd50801c2571595d15f4d9c339cbf774db6480b7626e08dfa: Status 404 returned error can't find the container with id aaca9b3a3f9d594fd50801c2571595d15f4d9c339cbf774db6480b7626e08dfa Jul 09 15:32:00.448468 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:32:00.448449 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-7b7744d957-k6dtg"] Jul 09 15:32:00.451924 ip-10-0-133-183 kubenswrapper[2573]: W0709 15:32:00.451900 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2c459d95_3f7c_4b06_b0ce_7e938d6c2c48.slice/crio-2ad64b24126926df756c833110e64b230207aa9caca1a49aa24e15a3eacc5258 WatchSource:0}: Error finding container 2ad64b24126926df756c833110e64b230207aa9caca1a49aa24e15a3eacc5258: Status 404 returned error can't find the container with id 2ad64b24126926df756c833110e64b230207aa9caca1a49aa24e15a3eacc5258 Jul 09 15:32:00.463286 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:32:00.463264 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/b9e43e24-97dc-464d-a445-2100e1b546a9-metrics-tls\") pod \"dns-default-ffqv6\" (UID: \"b9e43e24-97dc-464d-a445-2100e1b546a9\") " pod="openshift-dns/dns-default-ffqv6" Jul 09 15:32:00.463392 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:32:00.463349 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/81be5afd-69ba-4d11-95d0-efdb0f62bad0-registry-tls\") pod \"image-registry-7c768548b6-5tl98\" (UID: \"81be5afd-69ba-4d11-95d0-efdb0f62bad0\") " pod="openshift-image-registry/image-registry-7c768548b6-5tl98" Jul 09 15:32:00.463460 ip-10-0-133-183 kubenswrapper[2573]: E0709 15:32:00.463408 2573 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 09 15:32:00.463514 ip-10-0-133-183 kubenswrapper[2573]: E0709 15:32:00.463470 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b9e43e24-97dc-464d-a445-2100e1b546a9-metrics-tls podName:b9e43e24-97dc-464d-a445-2100e1b546a9 nodeName:}" failed. No retries permitted until 2026-07-09 15:32:04.463453812 +0000 UTC m=+46.184960660 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/b9e43e24-97dc-464d-a445-2100e1b546a9-metrics-tls") pod "dns-default-ffqv6" (UID: "b9e43e24-97dc-464d-a445-2100e1b546a9") : secret "dns-default-metrics-tls" not found Jul 09 15:32:00.463514 ip-10-0-133-183 kubenswrapper[2573]: E0709 15:32:00.463484 2573 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 09 15:32:00.463514 ip-10-0-133-183 kubenswrapper[2573]: E0709 15:32:00.463499 2573 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-7c768548b6-5tl98: secret "image-registry-tls" not found Jul 09 15:32:00.463653 ip-10-0-133-183 kubenswrapper[2573]: E0709 15:32:00.463583 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/81be5afd-69ba-4d11-95d0-efdb0f62bad0-registry-tls podName:81be5afd-69ba-4d11-95d0-efdb0f62bad0 nodeName:}" failed. No retries permitted until 2026-07-09 15:32:04.463536763 +0000 UTC m=+46.185043615 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/81be5afd-69ba-4d11-95d0-efdb0f62bad0-registry-tls") pod "image-registry-7c768548b6-5tl98" (UID: "81be5afd-69ba-4d11-95d0-efdb0f62bad0") : secret "image-registry-tls" not found Jul 09 15:32:00.611654 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:32:00.611418 2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator/migrator-7f59b54f6c-vj7jk"] Jul 09 15:32:00.629836 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:32:00.629806 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-7f59b54f6c-vj7jk"] Jul 09 15:32:00.630006 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:32:00.629945 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-7f59b54f6c-vj7jk" Jul 09 15:32:00.632804 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:32:00.632782 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"openshift-service-ca.crt\"" Jul 09 15:32:00.632945 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:32:00.632787 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-storage-version-migrator-sa-dockercfg-cpkq9\"" Jul 09 15:32:00.632945 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:32:00.632826 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-root-ca.crt\"" Jul 09 15:32:00.766291 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:32:00.766247 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hhpl8\" (UniqueName: \"kubernetes.io/projected/f4b336f8-f3d0-463d-952c-13622744cf40-kube-api-access-hhpl8\") pod \"migrator-7f59b54f6c-vj7jk\" (UID: \"f4b336f8-f3d0-463d-952c-13622744cf40\") " pod="openshift-kube-storage-version-migrator/migrator-7f59b54f6c-vj7jk" Jul 09 15:32:00.867477 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:32:00.867363 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hhpl8\" (UniqueName: \"kubernetes.io/projected/f4b336f8-f3d0-463d-952c-13622744cf40-kube-api-access-hhpl8\") pod \"migrator-7f59b54f6c-vj7jk\" (UID: \"f4b336f8-f3d0-463d-952c-13622744cf40\") " pod="openshift-kube-storage-version-migrator/migrator-7f59b54f6c-vj7jk" Jul 09 15:32:00.877509 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:32:00.877473 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hhpl8\" (UniqueName: \"kubernetes.io/projected/f4b336f8-f3d0-463d-952c-13622744cf40-kube-api-access-hhpl8\") pod \"migrator-7f59b54f6c-vj7jk\" (UID: \"f4b336f8-f3d0-463d-952c-13622744cf40\") " pod="openshift-kube-storage-version-migrator/migrator-7f59b54f6c-vj7jk" Jul 09 15:32:00.939634 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:32:00.939597 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-7f59b54f6c-vj7jk" Jul 09 15:32:01.037831 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:32:01.037785 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-7b7744d957-k6dtg" event={"ID":"2c459d95-3f7c-4b06-b0ce-7e938d6c2c48","Type":"ContainerStarted","Data":"2ad64b24126926df756c833110e64b230207aa9caca1a49aa24e15a3eacc5258"} Jul 09 15:32:01.039099 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:32:01.039069 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-6d4489575b-4x9ks" event={"ID":"48292f30-aa88-45e0-8bed-4c675ec638b6","Type":"ContainerStarted","Data":"aaca9b3a3f9d594fd50801c2571595d15f4d9c339cbf774db6480b7626e08dfa"} Jul 09 15:32:01.097590 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:32:01.097557 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-7f59b54f6c-vj7jk"] Jul 09 15:32:01.101389 ip-10-0-133-183 kubenswrapper[2573]: W0709 15:32:01.101362 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf4b336f8_f3d0_463d_952c_13622744cf40.slice/crio-54a891c34a311f4f1eb565c7bd2e293c7f6402b1f566be5234560c92da3632b4 WatchSource:0}: Error finding container 54a891c34a311f4f1eb565c7bd2e293c7f6402b1f566be5234560c92da3632b4: Status 404 returned error can't find the container with id 54a891c34a311f4f1eb565c7bd2e293c7f6402b1f566be5234560c92da3632b4 Jul 09 15:32:02.043854 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:32:02.043736 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-7f59b54f6c-vj7jk" event={"ID":"f4b336f8-f3d0-463d-952c-13622744cf40","Type":"ContainerStarted","Data":"54a891c34a311f4f1eb565c7bd2e293c7f6402b1f566be5234560c92da3632b4"} Jul 09 15:32:04.050348 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:32:04.050320 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-7f59b54f6c-vj7jk" event={"ID":"f4b336f8-f3d0-463d-952c-13622744cf40","Type":"ContainerStarted","Data":"f884dd5663d284754163ebd02bcfdcc63dbc5a6e96697b364fd3e69265337da7"} Jul 09 15:32:04.497271 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:32:04.497234 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/81be5afd-69ba-4d11-95d0-efdb0f62bad0-registry-tls\") pod \"image-registry-7c768548b6-5tl98\" (UID: \"81be5afd-69ba-4d11-95d0-efdb0f62bad0\") " pod="openshift-image-registry/image-registry-7c768548b6-5tl98" Jul 09 15:32:04.497445 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:32:04.497294 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/b9e43e24-97dc-464d-a445-2100e1b546a9-metrics-tls\") pod \"dns-default-ffqv6\" (UID: \"b9e43e24-97dc-464d-a445-2100e1b546a9\") " pod="openshift-dns/dns-default-ffqv6" Jul 09 15:32:04.497445 ip-10-0-133-183 kubenswrapper[2573]: E0709 15:32:04.497387 2573 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 09 15:32:04.497445 ip-10-0-133-183 kubenswrapper[2573]: E0709 15:32:04.497390 2573 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 09 15:32:04.497445 ip-10-0-133-183 kubenswrapper[2573]: E0709 15:32:04.497410 2573 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-7c768548b6-5tl98: secret "image-registry-tls" not found Jul 09 15:32:04.497445 ip-10-0-133-183 kubenswrapper[2573]: E0709 15:32:04.497442 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b9e43e24-97dc-464d-a445-2100e1b546a9-metrics-tls podName:b9e43e24-97dc-464d-a445-2100e1b546a9 nodeName:}" failed. No retries permitted until 2026-07-09 15:32:12.497427689 +0000 UTC m=+54.218934536 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/b9e43e24-97dc-464d-a445-2100e1b546a9-metrics-tls") pod "dns-default-ffqv6" (UID: "b9e43e24-97dc-464d-a445-2100e1b546a9") : secret "dns-default-metrics-tls" not found Jul 09 15:32:04.497597 ip-10-0-133-183 kubenswrapper[2573]: E0709 15:32:04.497456 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/81be5afd-69ba-4d11-95d0-efdb0f62bad0-registry-tls podName:81be5afd-69ba-4d11-95d0-efdb0f62bad0 nodeName:}" failed. No retries permitted until 2026-07-09 15:32:12.497450162 +0000 UTC m=+54.218957008 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/81be5afd-69ba-4d11-95d0-efdb0f62bad0-registry-tls") pod "image-registry-7c768548b6-5tl98" (UID: "81be5afd-69ba-4d11-95d0-efdb0f62bad0") : secret "image-registry-tls" not found Jul 09 15:32:05.055602 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:32:05.055563 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-7f59b54f6c-vj7jk" event={"ID":"f4b336f8-f3d0-463d-952c-13622744cf40","Type":"ContainerStarted","Data":"87a6aab9de0950ee6b422f55024cbc2bb3a51d9d723f81578622dbcaf150cd29"} Jul 09 15:32:05.077075 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:32:05.077014 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator/migrator-7f59b54f6c-vj7jk" podStartSLOduration=2.339182091 podStartE2EDuration="5.076991629s" totalImagesPullingTime="2.737809538s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 15:32:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 15:32:05.074962974 +0000 UTC m=+46.796469858" watchObservedRunningTime="2026-07-09 15:32:05.076991629 +0000 UTC m=+46.798498499" Jul 09 15:32:08.065186 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:32:08.065141 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-7b7744d957-k6dtg" event={"ID":"2c459d95-3f7c-4b06-b0ce-7e938d6c2c48","Type":"ContainerStarted","Data":"7ae8122f61a029970a1df53bfee48ac58e069eaabf91848b907221ed6dda2ac3"} Jul 09 15:32:08.065665 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:32:08.065300 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-7b7744d957-k6dtg" Jul 09 15:32:08.066784 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:32:08.066748 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-6d4489575b-4x9ks" event={"ID":"48292f30-aa88-45e0-8bed-4c675ec638b6","Type":"ContainerStarted","Data":"f5c4debcb0817fa2f77272f2daaf4078d897d1c3ab4c348a4612a28616fc7119"} Jul 09 15:32:08.067341 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:32:08.067325 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-7b7744d957-k6dtg" Jul 09 15:32:08.084625 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:32:08.084577 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-7b7744d957-k6dtg" podStartSLOduration=2.132019222 podStartE2EDuration="9.084563506s" totalImagesPullingTime="6.952544284s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 15:31:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 15:32:08.083760675 +0000 UTC m=+49.805267556" watchObservedRunningTime="2026-07-09 15:32:08.084563506 +0000 UTC m=+49.806070374" Jul 09 15:32:08.117000 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:32:08.116951 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-6d4489575b-4x9ks" podStartSLOduration=2.15547288 podStartE2EDuration="9.116935413s" totalImagesPullingTime="6.961462533s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 15:31:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 15:32:08.116137207 +0000 UTC m=+49.837644075" watchObservedRunningTime="2026-07-09 15:32:08.116935413 +0000 UTC m=+49.838442281" Jul 09 15:32:12.560472 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:32:12.560434 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/81be5afd-69ba-4d11-95d0-efdb0f62bad0-registry-tls\") pod \"image-registry-7c768548b6-5tl98\" (UID: \"81be5afd-69ba-4d11-95d0-efdb0f62bad0\") " pod="openshift-image-registry/image-registry-7c768548b6-5tl98" Jul 09 15:32:12.560977 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:32:12.560531 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/b9e43e24-97dc-464d-a445-2100e1b546a9-metrics-tls\") pod \"dns-default-ffqv6\" (UID: \"b9e43e24-97dc-464d-a445-2100e1b546a9\") " pod="openshift-dns/dns-default-ffqv6" Jul 09 15:32:12.563221 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:32:12.563192 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/b9e43e24-97dc-464d-a445-2100e1b546a9-metrics-tls\") pod \"dns-default-ffqv6\" (UID: \"b9e43e24-97dc-464d-a445-2100e1b546a9\") " pod="openshift-dns/dns-default-ffqv6" Jul 09 15:32:12.563376 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:32:12.563354 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/81be5afd-69ba-4d11-95d0-efdb0f62bad0-registry-tls\") pod \"image-registry-7c768548b6-5tl98\" (UID: \"81be5afd-69ba-4d11-95d0-efdb0f62bad0\") " pod="openshift-image-registry/image-registry-7c768548b6-5tl98" Jul 09 15:32:12.587269 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:32:12.587230 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-7c768548b6-5tl98" Jul 09 15:32:12.601583 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:32:12.601549 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-ffqv6" Jul 09 15:32:12.780442 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:32:12.780408 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-7c768548b6-5tl98"] Jul 09 15:32:12.784755 ip-10-0-133-183 kubenswrapper[2573]: W0709 15:32:12.784724 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod81be5afd_69ba_4d11_95d0_efdb0f62bad0.slice/crio-ca80b44581823c2bc67be7fdb9bba88189543a7633d48280f66d43896c550b93 WatchSource:0}: Error finding container ca80b44581823c2bc67be7fdb9bba88189543a7633d48280f66d43896c550b93: Status 404 returned error can't find the container with id ca80b44581823c2bc67be7fdb9bba88189543a7633d48280f66d43896c550b93 Jul 09 15:32:12.798272 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:32:12.798238 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-ffqv6"] Jul 09 15:32:12.802000 ip-10-0-133-183 kubenswrapper[2573]: W0709 15:32:12.801972 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb9e43e24_97dc_464d_a445_2100e1b546a9.slice/crio-6a6b465d42aef04e01dfd640bdbd0f53b53e897c6aaf108d203a3ff1dfac4756 WatchSource:0}: Error finding container 6a6b465d42aef04e01dfd640bdbd0f53b53e897c6aaf108d203a3ff1dfac4756: Status 404 returned error can't find the container with id 6a6b465d42aef04e01dfd640bdbd0f53b53e897c6aaf108d203a3ff1dfac4756 Jul 09 15:32:13.086916 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:32:13.084916 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-7c768548b6-5tl98" event={"ID":"81be5afd-69ba-4d11-95d0-efdb0f62bad0","Type":"ContainerStarted","Data":"5e962fa8b5472577d1c52638b944f0aab609f29ef0212fb93138e09e80a02c92"} Jul 09 15:32:13.086916 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:32:13.085003 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-7c768548b6-5tl98" event={"ID":"81be5afd-69ba-4d11-95d0-efdb0f62bad0","Type":"ContainerStarted","Data":"ca80b44581823c2bc67be7fdb9bba88189543a7633d48280f66d43896c550b93"} Jul 09 15:32:13.089485 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:32:13.089450 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-ffqv6" event={"ID":"b9e43e24-97dc-464d-a445-2100e1b546a9","Type":"ContainerStarted","Data":"6a6b465d42aef04e01dfd640bdbd0f53b53e897c6aaf108d203a3ff1dfac4756"} Jul 09 15:32:13.109413 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:32:13.109359 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/image-registry-7c768548b6-5tl98" podStartSLOduration=55.109342717 podStartE2EDuration="55.109342717s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 15:31:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 15:32:13.108557815 +0000 UTC m=+54.830064707" watchObservedRunningTime="2026-07-09 15:32:13.109342717 +0000 UTC m=+54.830849577" Jul 09 15:32:14.092713 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:32:14.092680 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-7c768548b6-5tl98" Jul 09 15:32:15.097718 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:32:15.097619 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-ffqv6" event={"ID":"b9e43e24-97dc-464d-a445-2100e1b546a9","Type":"ContainerStarted","Data":"299057c8efdc9ceacb87b47575e40f2a27068212a7548cd712a0e3e413dce4c9"} Jul 09 15:32:15.097718 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:32:15.097687 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-ffqv6" event={"ID":"b9e43e24-97dc-464d-a445-2100e1b546a9","Type":"ContainerStarted","Data":"17d91e84bfe8caaf1d4a65506e2c5dfc7fd5142e2612dc4d2eb300a55bddd28c"} Jul 09 15:32:15.118303 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:32:15.118252 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-ffqv6" podStartSLOduration=17.07343999 podStartE2EDuration="19.118235286s" totalImagesPullingTime="2.044795296s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 15:31:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 15:32:15.118142826 +0000 UTC m=+56.839649695" watchObservedRunningTime="2026-07-09 15:32:15.118235286 +0000 UTC m=+56.839742154" Jul 09 15:32:16.100929 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:32:16.100890 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-ffqv6" Jul 09 15:32:22.013524 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:32:22.013491 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-b7b9v" Jul 09 15:32:24.552567 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:32:24.552528 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/d9ea5934-356a-47e5-9fc6-1ae4f39f4e19-original-pull-secret\") pod \"global-pull-secret-syncer-h2cs6\" (UID: \"d9ea5934-356a-47e5-9fc6-1ae4f39f4e19\") " pod="kube-system/global-pull-secret-syncer-h2cs6" Jul 09 15:32:24.552567 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:32:24.552574 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8e56f4e6-cbec-422f-9505-c7c6654159d9-metrics-certs\") pod \"network-metrics-daemon-8f2p4\" (UID: \"8e56f4e6-cbec-422f-9505-c7c6654159d9\") " pod="openshift-multus/network-metrics-daemon-8f2p4" Jul 09 15:32:24.553049 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:32:24.552597 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l6q9n\" (UniqueName: \"kubernetes.io/projected/e8de13cc-b0e0-4a54-bd44-3ba73cbf2bf2-kube-api-access-l6q9n\") pod \"network-check-target-sdbbl\" (UID: \"e8de13cc-b0e0-4a54-bd44-3ba73cbf2bf2\") " pod="openshift-network-diagnostics/network-check-target-sdbbl" Jul 09 15:32:24.556193 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:32:24.556173 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Jul 09 15:32:24.557512 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:32:24.557493 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Jul 09 15:32:24.557565 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:32:24.557494 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Jul 09 15:32:24.565670 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:32:24.565642 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/d9ea5934-356a-47e5-9fc6-1ae4f39f4e19-original-pull-secret\") pod \"global-pull-secret-syncer-h2cs6\" (UID: \"d9ea5934-356a-47e5-9fc6-1ae4f39f4e19\") " pod="kube-system/global-pull-secret-syncer-h2cs6" Jul 09 15:32:24.565670 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:32:24.565657 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8e56f4e6-cbec-422f-9505-c7c6654159d9-metrics-certs\") pod \"network-metrics-daemon-8f2p4\" (UID: \"8e56f4e6-cbec-422f-9505-c7c6654159d9\") " pod="openshift-multus/network-metrics-daemon-8f2p4" Jul 09 15:32:24.570781 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:32:24.570763 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Jul 09 15:32:24.576423 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:32:24.576406 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-l6q9n\" (UniqueName: \"kubernetes.io/projected/e8de13cc-b0e0-4a54-bd44-3ba73cbf2bf2-kube-api-access-l6q9n\") pod \"network-check-target-sdbbl\" (UID: \"e8de13cc-b0e0-4a54-bd44-3ba73cbf2bf2\") " pod="openshift-network-diagnostics/network-check-target-sdbbl" Jul 09 15:32:24.721632 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:32:24.721598 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-h2cs6" Jul 09 15:32:24.722009 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:32:24.721992 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-bnkz7\"" Jul 09 15:32:24.726064 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:32:24.726039 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-sdbbl" Jul 09 15:32:24.729047 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:32:24.729026 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-rsq6f\"" Jul 09 15:32:24.736112 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:32:24.736056 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-8f2p4" Jul 09 15:32:24.879856 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:32:24.879826 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-sdbbl"] Jul 09 15:32:24.882654 ip-10-0-133-183 kubenswrapper[2573]: W0709 15:32:24.882627 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode8de13cc_b0e0_4a54_bd44_3ba73cbf2bf2.slice/crio-0edf3cf7a5d37008b17631049c20cdbc1f02594b5e983939f480d1962e1065ea WatchSource:0}: Error finding container 0edf3cf7a5d37008b17631049c20cdbc1f02594b5e983939f480d1962e1065ea: Status 404 returned error can't find the container with id 0edf3cf7a5d37008b17631049c20cdbc1f02594b5e983939f480d1962e1065ea Jul 09 15:32:25.104258 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:32:25.104174 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-8f2p4"] Jul 09 15:32:25.106784 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:32:25.106735 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-h2cs6"] Jul 09 15:32:25.107355 ip-10-0-133-183 kubenswrapper[2573]: W0709 15:32:25.107321 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8e56f4e6_cbec_422f_9505_c7c6654159d9.slice/crio-bf5c15533c035a10ac33b3b63fecc055191fe62dcab9c89a4965509a10574e1e WatchSource:0}: Error finding container bf5c15533c035a10ac33b3b63fecc055191fe62dcab9c89a4965509a10574e1e: Status 404 returned error can't find the container with id bf5c15533c035a10ac33b3b63fecc055191fe62dcab9c89a4965509a10574e1e Jul 09 15:32:25.111712 ip-10-0-133-183 kubenswrapper[2573]: W0709 15:32:25.111684 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd9ea5934_356a_47e5_9fc6_1ae4f39f4e19.slice/crio-55c366856487a72b7d1069a6c313dbd7cd78b12f386b6b457961643a49fd574b WatchSource:0}: Error finding container 55c366856487a72b7d1069a6c313dbd7cd78b12f386b6b457961643a49fd574b: Status 404 returned error can't find the container with id 55c366856487a72b7d1069a6c313dbd7cd78b12f386b6b457961643a49fd574b Jul 09 15:32:25.128762 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:32:25.128732 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-h2cs6" event={"ID":"d9ea5934-356a-47e5-9fc6-1ae4f39f4e19","Type":"ContainerStarted","Data":"55c366856487a72b7d1069a6c313dbd7cd78b12f386b6b457961643a49fd574b"} Jul 09 15:32:25.129940 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:32:25.129906 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-8f2p4" event={"ID":"8e56f4e6-cbec-422f-9505-c7c6654159d9","Type":"ContainerStarted","Data":"bf5c15533c035a10ac33b3b63fecc055191fe62dcab9c89a4965509a10574e1e"} Jul 09 15:32:25.130991 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:32:25.130970 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-sdbbl" event={"ID":"e8de13cc-b0e0-4a54-bd44-3ba73cbf2bf2","Type":"ContainerStarted","Data":"0edf3cf7a5d37008b17631049c20cdbc1f02594b5e983939f480d1962e1065ea"} Jul 09 15:32:26.105394 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:32:26.105360 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-ffqv6" Jul 09 15:32:27.138454 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:32:27.138367 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-8f2p4" event={"ID":"8e56f4e6-cbec-422f-9505-c7c6654159d9","Type":"ContainerStarted","Data":"bb4b6de076555a6c68dbc41f141dcac90d1b198b05d8566b4c54afadc94ad45c"} Jul 09 15:32:27.138454 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:32:27.138434 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-8f2p4" event={"ID":"8e56f4e6-cbec-422f-9505-c7c6654159d9","Type":"ContainerStarted","Data":"55c9e567b3f0a375d0cea32efd952d6ddfe44111bbbb7f1794ce6c831e0ce21a"} Jul 09 15:32:27.168536 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:32:27.168479 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-8f2p4" podStartSLOduration=67.847530976 podStartE2EDuration="1m9.16846101s" totalImagesPullingTime="1.320930034s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 15:31:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 15:32:27.168014783 +0000 UTC m=+68.889521652" watchObservedRunningTime="2026-07-09 15:32:27.16846101 +0000 UTC m=+68.889967879" Jul 09 15:32:27.845442 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:32:27.845410 2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-7c768548b6-5tl98"] Jul 09 15:32:27.853627 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:32:27.853593 2573 patch_prober.go:28] interesting pod/image-registry-7c768548b6-5tl98 container/registry namespace/openshift-image-registry: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body={"errors":[{"code":"UNAVAILABLE","message":"service unavailable","detail":"health check failed: please see /debug/health"}]} Jul 09 15:32:27.853827 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:32:27.853653 2573 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-image-registry/image-registry-7c768548b6-5tl98" podUID="81be5afd-69ba-4d11-95d0-efdb0f62bad0" containerName="registry" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 09 15:32:30.150315 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:32:30.150281 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-sdbbl" event={"ID":"e8de13cc-b0e0-4a54-bd44-3ba73cbf2bf2","Type":"ContainerStarted","Data":"4d4041deda7a98172823cb764373a93349ccf39bd9d37bff7107564fd1ba2945"} Jul 09 15:32:30.150893 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:32:30.150382 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-sdbbl" Jul 09 15:32:30.170597 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:32:30.170531 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-sdbbl" podStartSLOduration=67.837779476 podStartE2EDuration="1m12.170508457s" totalImagesPullingTime="4.332728981s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 15:31:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 15:32:30.16927241 +0000 UTC m=+71.890779277" watchObservedRunningTime="2026-07-09 15:32:30.170508457 +0000 UTC m=+71.892015328" Jul 09 15:32:32.992980 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:32:32.992943 2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-5kk64"] Jul 09 15:32:32.995124 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:32:32.995103 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-5kk64" Jul 09 15:32:32.997691 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:32:32.997666 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\"" Jul 09 15:32:32.997810 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:32:32.997748 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\"" Jul 09 15:32:32.998984 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:32:32.998967 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\"" Jul 09 15:32:32.998984 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:32:32.998981 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\"" Jul 09 15:32:32.999147 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:32:32.998965 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-f9q7c\"" Jul 09 15:32:33.004278 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:32:33.004255 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-5kk64"] Jul 09 15:32:33.120697 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:32:33.120660 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/8c2fc9e6-25b9-4d1d-9ea0-c6de5f1367e2-data-volume\") pod \"insights-runtime-extractor-5kk64\" (UID: \"8c2fc9e6-25b9-4d1d-9ea0-c6de5f1367e2\") " pod="openshift-insights/insights-runtime-extractor-5kk64" Jul 09 15:32:33.120897 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:32:33.120710 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cz28t\" (UniqueName: \"kubernetes.io/projected/8c2fc9e6-25b9-4d1d-9ea0-c6de5f1367e2-kube-api-access-cz28t\") pod \"insights-runtime-extractor-5kk64\" (UID: \"8c2fc9e6-25b9-4d1d-9ea0-c6de5f1367e2\") " pod="openshift-insights/insights-runtime-extractor-5kk64" Jul 09 15:32:33.120897 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:32:33.120731 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/8c2fc9e6-25b9-4d1d-9ea0-c6de5f1367e2-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-5kk64\" (UID: \"8c2fc9e6-25b9-4d1d-9ea0-c6de5f1367e2\") " pod="openshift-insights/insights-runtime-extractor-5kk64" Jul 09 15:32:33.120897 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:32:33.120755 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/8c2fc9e6-25b9-4d1d-9ea0-c6de5f1367e2-crio-socket\") pod \"insights-runtime-extractor-5kk64\" (UID: \"8c2fc9e6-25b9-4d1d-9ea0-c6de5f1367e2\") " pod="openshift-insights/insights-runtime-extractor-5kk64" Jul 09 15:32:33.120897 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:32:33.120817 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/8c2fc9e6-25b9-4d1d-9ea0-c6de5f1367e2-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-5kk64\" (UID: \"8c2fc9e6-25b9-4d1d-9ea0-c6de5f1367e2\") " pod="openshift-insights/insights-runtime-extractor-5kk64" Jul 09 15:32:33.162087 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:32:33.162056 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-h2cs6" event={"ID":"d9ea5934-356a-47e5-9fc6-1ae4f39f4e19","Type":"ContainerStarted","Data":"d2bfcf1ff311e3b23966083e9ec165cea58dd5c0848bd6ba7461382b07485ef4"} Jul 09 15:32:33.181095 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:32:33.181053 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-h2cs6" podStartSLOduration=67.85899614 podStartE2EDuration="1m15.181036887s" totalImagesPullingTime="7.322040747s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-09 15:31:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 15:32:33.180073023 +0000 UTC m=+74.901579894" watchObservedRunningTime="2026-07-09 15:32:33.181036887 +0000 UTC m=+74.902543756" Jul 09 15:32:33.222150 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:32:33.222115 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cz28t\" (UniqueName: \"kubernetes.io/projected/8c2fc9e6-25b9-4d1d-9ea0-c6de5f1367e2-kube-api-access-cz28t\") pod \"insights-runtime-extractor-5kk64\" (UID: \"8c2fc9e6-25b9-4d1d-9ea0-c6de5f1367e2\") " pod="openshift-insights/insights-runtime-extractor-5kk64" Jul 09 15:32:33.222150 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:32:33.222150 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/8c2fc9e6-25b9-4d1d-9ea0-c6de5f1367e2-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-5kk64\" (UID: \"8c2fc9e6-25b9-4d1d-9ea0-c6de5f1367e2\") " pod="openshift-insights/insights-runtime-extractor-5kk64" Jul 09 15:32:33.222337 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:32:33.222168 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/8c2fc9e6-25b9-4d1d-9ea0-c6de5f1367e2-crio-socket\") pod \"insights-runtime-extractor-5kk64\" (UID: \"8c2fc9e6-25b9-4d1d-9ea0-c6de5f1367e2\") " pod="openshift-insights/insights-runtime-extractor-5kk64" Jul 09 15:32:33.222337 ip-10-0-133-183 kubenswrapper[2573]: E0709 15:32:33.222247 2573 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 09 15:32:33.222337 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:32:33.222277 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/8c2fc9e6-25b9-4d1d-9ea0-c6de5f1367e2-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-5kk64\" (UID: \"8c2fc9e6-25b9-4d1d-9ea0-c6de5f1367e2\") " pod="openshift-insights/insights-runtime-extractor-5kk64" Jul 09 15:32:33.222337 ip-10-0-133-183 kubenswrapper[2573]: E0709 15:32:33.222326 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/8c2fc9e6-25b9-4d1d-9ea0-c6de5f1367e2-kube-rbac-proxy-cm podName:8c2fc9e6-25b9-4d1d-9ea0-c6de5f1367e2 nodeName:}" failed. No retries permitted until 2026-07-09 15:32:33.722308146 +0000 UTC m=+75.443814993 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/8c2fc9e6-25b9-4d1d-9ea0-c6de5f1367e2-kube-rbac-proxy-cm") pod "insights-runtime-extractor-5kk64" (UID: "8c2fc9e6-25b9-4d1d-9ea0-c6de5f1367e2") : configmap "kube-rbac-proxy" not found Jul 09 15:32:33.222482 ip-10-0-133-183 kubenswrapper[2573]: E0709 15:32:33.222367 2573 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 09 15:32:33.222482 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:32:33.222372 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/8c2fc9e6-25b9-4d1d-9ea0-c6de5f1367e2-data-volume\") pod \"insights-runtime-extractor-5kk64\" (UID: \"8c2fc9e6-25b9-4d1d-9ea0-c6de5f1367e2\") " pod="openshift-insights/insights-runtime-extractor-5kk64" Jul 09 15:32:33.222482 ip-10-0-133-183 kubenswrapper[2573]: E0709 15:32:33.222424 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8c2fc9e6-25b9-4d1d-9ea0-c6de5f1367e2-insights-runtime-extractor-tls podName:8c2fc9e6-25b9-4d1d-9ea0-c6de5f1367e2 nodeName:}" failed. No retries permitted until 2026-07-09 15:32:33.722406047 +0000 UTC m=+75.443912902 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/8c2fc9e6-25b9-4d1d-9ea0-c6de5f1367e2-insights-runtime-extractor-tls") pod "insights-runtime-extractor-5kk64" (UID: "8c2fc9e6-25b9-4d1d-9ea0-c6de5f1367e2") : secret "insights-runtime-extractor-tls" not found Jul 09 15:32:33.222482 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:32:33.222451 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/8c2fc9e6-25b9-4d1d-9ea0-c6de5f1367e2-crio-socket\") pod \"insights-runtime-extractor-5kk64\" (UID: \"8c2fc9e6-25b9-4d1d-9ea0-c6de5f1367e2\") " pod="openshift-insights/insights-runtime-extractor-5kk64" Jul 09 15:32:33.222713 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:32:33.222697 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/8c2fc9e6-25b9-4d1d-9ea0-c6de5f1367e2-data-volume\") pod \"insights-runtime-extractor-5kk64\" (UID: \"8c2fc9e6-25b9-4d1d-9ea0-c6de5f1367e2\") " pod="openshift-insights/insights-runtime-extractor-5kk64" Jul 09 15:32:33.231985 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:32:33.231963 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cz28t\" (UniqueName: \"kubernetes.io/projected/8c2fc9e6-25b9-4d1d-9ea0-c6de5f1367e2-kube-api-access-cz28t\") pod \"insights-runtime-extractor-5kk64\" (UID: \"8c2fc9e6-25b9-4d1d-9ea0-c6de5f1367e2\") " pod="openshift-insights/insights-runtime-extractor-5kk64" Jul 09 15:32:33.726926 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:32:33.726846 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/8c2fc9e6-25b9-4d1d-9ea0-c6de5f1367e2-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-5kk64\" (UID: \"8c2fc9e6-25b9-4d1d-9ea0-c6de5f1367e2\") " pod="openshift-insights/insights-runtime-extractor-5kk64" Jul 09 15:32:33.727136 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:32:33.726940 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/8c2fc9e6-25b9-4d1d-9ea0-c6de5f1367e2-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-5kk64\" (UID: \"8c2fc9e6-25b9-4d1d-9ea0-c6de5f1367e2\") " pod="openshift-insights/insights-runtime-extractor-5kk64" Jul 09 15:32:33.727136 ip-10-0-133-183 kubenswrapper[2573]: E0709 15:32:33.726986 2573 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 09 15:32:33.727136 ip-10-0-133-183 kubenswrapper[2573]: E0709 15:32:33.727065 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/8c2fc9e6-25b9-4d1d-9ea0-c6de5f1367e2-kube-rbac-proxy-cm podName:8c2fc9e6-25b9-4d1d-9ea0-c6de5f1367e2 nodeName:}" failed. No retries permitted until 2026-07-09 15:32:34.727044923 +0000 UTC m=+76.448551789 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/8c2fc9e6-25b9-4d1d-9ea0-c6de5f1367e2-kube-rbac-proxy-cm") pod "insights-runtime-extractor-5kk64" (UID: "8c2fc9e6-25b9-4d1d-9ea0-c6de5f1367e2") : configmap "kube-rbac-proxy" not found Jul 09 15:32:33.727136 ip-10-0-133-183 kubenswrapper[2573]: E0709 15:32:33.727094 2573 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 09 15:32:33.727136 ip-10-0-133-183 kubenswrapper[2573]: E0709 15:32:33.727136 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8c2fc9e6-25b9-4d1d-9ea0-c6de5f1367e2-insights-runtime-extractor-tls podName:8c2fc9e6-25b9-4d1d-9ea0-c6de5f1367e2 nodeName:}" failed. No retries permitted until 2026-07-09 15:32:34.727124899 +0000 UTC m=+76.448631746 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/8c2fc9e6-25b9-4d1d-9ea0-c6de5f1367e2-insights-runtime-extractor-tls") pod "insights-runtime-extractor-5kk64" (UID: "8c2fc9e6-25b9-4d1d-9ea0-c6de5f1367e2") : secret "insights-runtime-extractor-tls" not found Jul 09 15:32:34.733684 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:32:34.733639 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/8c2fc9e6-25b9-4d1d-9ea0-c6de5f1367e2-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-5kk64\" (UID: \"8c2fc9e6-25b9-4d1d-9ea0-c6de5f1367e2\") " pod="openshift-insights/insights-runtime-extractor-5kk64" Jul 09 15:32:34.733684 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:32:34.733684 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/8c2fc9e6-25b9-4d1d-9ea0-c6de5f1367e2-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-5kk64\" (UID: \"8c2fc9e6-25b9-4d1d-9ea0-c6de5f1367e2\") " pod="openshift-insights/insights-runtime-extractor-5kk64" Jul 09 15:32:34.734150 ip-10-0-133-183 kubenswrapper[2573]: E0709 15:32:34.733770 2573 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 09 15:32:34.734150 ip-10-0-133-183 kubenswrapper[2573]: E0709 15:32:34.733795 2573 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 09 15:32:34.734150 ip-10-0-133-183 kubenswrapper[2573]: E0709 15:32:34.733839 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/8c2fc9e6-25b9-4d1d-9ea0-c6de5f1367e2-kube-rbac-proxy-cm podName:8c2fc9e6-25b9-4d1d-9ea0-c6de5f1367e2 nodeName:}" failed. No retries permitted until 2026-07-09 15:32:36.733822336 +0000 UTC m=+78.455329187 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/8c2fc9e6-25b9-4d1d-9ea0-c6de5f1367e2-kube-rbac-proxy-cm") pod "insights-runtime-extractor-5kk64" (UID: "8c2fc9e6-25b9-4d1d-9ea0-c6de5f1367e2") : configmap "kube-rbac-proxy" not found Jul 09 15:32:34.734150 ip-10-0-133-183 kubenswrapper[2573]: E0709 15:32:34.733853 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8c2fc9e6-25b9-4d1d-9ea0-c6de5f1367e2-insights-runtime-extractor-tls podName:8c2fc9e6-25b9-4d1d-9ea0-c6de5f1367e2 nodeName:}" failed. No retries permitted until 2026-07-09 15:32:36.733846783 +0000 UTC m=+78.455353643 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/8c2fc9e6-25b9-4d1d-9ea0-c6de5f1367e2-insights-runtime-extractor-tls") pod "insights-runtime-extractor-5kk64" (UID: "8c2fc9e6-25b9-4d1d-9ea0-c6de5f1367e2") : secret "insights-runtime-extractor-tls" not found Jul 09 15:32:35.566512 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:32:35.566475 2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-d6c6bcfbb-fjcg2"] Jul 09 15:32:35.568947 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:32:35.568923 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-d6c6bcfbb-fjcg2" Jul 09 15:32:35.574768 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:32:35.574738 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\"" Jul 09 15:32:35.574768 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:32:35.574749 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\"" Jul 09 15:32:35.575674 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:32:35.575655 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\"" Jul 09 15:32:35.576356 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:32:35.576339 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\"" Jul 09 15:32:35.576444 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:32:35.576358 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\"" Jul 09 15:32:35.576444 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:32:35.576380 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-l8swt\"" Jul 09 15:32:35.577052 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:32:35.577037 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\"" Jul 09 15:32:35.577619 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:32:35.577601 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\"" Jul 09 15:32:35.585117 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:32:35.585094 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"trusted-ca-bundle\"" Jul 09 15:32:35.603854 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:32:35.603826 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-d6c6bcfbb-fjcg2"] Jul 09 15:32:35.639180 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:32:35.639147 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/fc654672-12df-4a71-827f-1951234b78e8-console-serving-cert\") pod \"console-d6c6bcfbb-fjcg2\" (UID: \"fc654672-12df-4a71-827f-1951234b78e8\") " pod="openshift-console/console-d6c6bcfbb-fjcg2" Jul 09 15:32:35.639180 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:32:35.639186 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7hh2b\" (UniqueName: \"kubernetes.io/projected/fc654672-12df-4a71-827f-1951234b78e8-kube-api-access-7hh2b\") pod \"console-d6c6bcfbb-fjcg2\" (UID: \"fc654672-12df-4a71-827f-1951234b78e8\") " pod="openshift-console/console-d6c6bcfbb-fjcg2" Jul 09 15:32:35.639365 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:32:35.639242 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/fc654672-12df-4a71-827f-1951234b78e8-console-config\") pod \"console-d6c6bcfbb-fjcg2\" (UID: \"fc654672-12df-4a71-827f-1951234b78e8\") " pod="openshift-console/console-d6c6bcfbb-fjcg2" Jul 09 15:32:35.639365 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:32:35.639291 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/fc654672-12df-4a71-827f-1951234b78e8-console-oauth-config\") pod \"console-d6c6bcfbb-fjcg2\" (UID: \"fc654672-12df-4a71-827f-1951234b78e8\") " pod="openshift-console/console-d6c6bcfbb-fjcg2" Jul 09 15:32:35.639365 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:32:35.639307 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/fc654672-12df-4a71-827f-1951234b78e8-service-ca\") pod \"console-d6c6bcfbb-fjcg2\" (UID: \"fc654672-12df-4a71-827f-1951234b78e8\") " pod="openshift-console/console-d6c6bcfbb-fjcg2" Jul 09 15:32:35.639365 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:32:35.639340 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/fc654672-12df-4a71-827f-1951234b78e8-trusted-ca-bundle\") pod \"console-d6c6bcfbb-fjcg2\" (UID: \"fc654672-12df-4a71-827f-1951234b78e8\") " pod="openshift-console/console-d6c6bcfbb-fjcg2" Jul 09 15:32:35.639483 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:32:35.639365 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/fc654672-12df-4a71-827f-1951234b78e8-oauth-serving-cert\") pod \"console-d6c6bcfbb-fjcg2\" (UID: \"fc654672-12df-4a71-827f-1951234b78e8\") " pod="openshift-console/console-d6c6bcfbb-fjcg2" Jul 09 15:32:35.739692 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:32:35.739659 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/fc654672-12df-4a71-827f-1951234b78e8-console-serving-cert\") pod \"console-d6c6bcfbb-fjcg2\" (UID: \"fc654672-12df-4a71-827f-1951234b78e8\") " pod="openshift-console/console-d6c6bcfbb-fjcg2" Jul 09 15:32:35.740109 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:32:35.739698 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7hh2b\" (UniqueName: \"kubernetes.io/projected/fc654672-12df-4a71-827f-1951234b78e8-kube-api-access-7hh2b\") pod \"console-d6c6bcfbb-fjcg2\" (UID: \"fc654672-12df-4a71-827f-1951234b78e8\") " pod="openshift-console/console-d6c6bcfbb-fjcg2" Jul 09 15:32:35.740109 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:32:35.739733 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/fc654672-12df-4a71-827f-1951234b78e8-console-config\") pod \"console-d6c6bcfbb-fjcg2\" (UID: \"fc654672-12df-4a71-827f-1951234b78e8\") " pod="openshift-console/console-d6c6bcfbb-fjcg2" Jul 09 15:32:35.740109 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:32:35.739756 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/fc654672-12df-4a71-827f-1951234b78e8-console-oauth-config\") pod \"console-d6c6bcfbb-fjcg2\" (UID: \"fc654672-12df-4a71-827f-1951234b78e8\") " pod="openshift-console/console-d6c6bcfbb-fjcg2" Jul 09 15:32:35.740109 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:32:35.739771 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/fc654672-12df-4a71-827f-1951234b78e8-service-ca\") pod \"console-d6c6bcfbb-fjcg2\" (UID: \"fc654672-12df-4a71-827f-1951234b78e8\") " pod="openshift-console/console-d6c6bcfbb-fjcg2" Jul 09 15:32:35.740109 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:32:35.739804 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/fc654672-12df-4a71-827f-1951234b78e8-trusted-ca-bundle\") pod \"console-d6c6bcfbb-fjcg2\" (UID: \"fc654672-12df-4a71-827f-1951234b78e8\") " pod="openshift-console/console-d6c6bcfbb-fjcg2" Jul 09 15:32:35.740109 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:32:35.739837 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/fc654672-12df-4a71-827f-1951234b78e8-oauth-serving-cert\") pod \"console-d6c6bcfbb-fjcg2\" (UID: \"fc654672-12df-4a71-827f-1951234b78e8\") " pod="openshift-console/console-d6c6bcfbb-fjcg2" Jul 09 15:32:35.740611 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:32:35.740589 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/fc654672-12df-4a71-827f-1951234b78e8-oauth-serving-cert\") pod \"console-d6c6bcfbb-fjcg2\" (UID: \"fc654672-12df-4a71-827f-1951234b78e8\") " pod="openshift-console/console-d6c6bcfbb-fjcg2" Jul 09 15:32:35.740705 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:32:35.740681 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/fc654672-12df-4a71-827f-1951234b78e8-service-ca\") pod \"console-d6c6bcfbb-fjcg2\" (UID: \"fc654672-12df-4a71-827f-1951234b78e8\") " pod="openshift-console/console-d6c6bcfbb-fjcg2" Jul 09 15:32:35.740770 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:32:35.740728 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/fc654672-12df-4a71-827f-1951234b78e8-console-config\") pod \"console-d6c6bcfbb-fjcg2\" (UID: \"fc654672-12df-4a71-827f-1951234b78e8\") " pod="openshift-console/console-d6c6bcfbb-fjcg2" Jul 09 15:32:35.740833 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:32:35.740804 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/fc654672-12df-4a71-827f-1951234b78e8-trusted-ca-bundle\") pod \"console-d6c6bcfbb-fjcg2\" (UID: \"fc654672-12df-4a71-827f-1951234b78e8\") " pod="openshift-console/console-d6c6bcfbb-fjcg2" Jul 09 15:32:35.742244 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:32:35.742222 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/fc654672-12df-4a71-827f-1951234b78e8-console-oauth-config\") pod \"console-d6c6bcfbb-fjcg2\" (UID: \"fc654672-12df-4a71-827f-1951234b78e8\") " pod="openshift-console/console-d6c6bcfbb-fjcg2" Jul 09 15:32:35.742537 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:32:35.742517 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/fc654672-12df-4a71-827f-1951234b78e8-console-serving-cert\") pod \"console-d6c6bcfbb-fjcg2\" (UID: \"fc654672-12df-4a71-827f-1951234b78e8\") " pod="openshift-console/console-d6c6bcfbb-fjcg2" Jul 09 15:32:35.759109 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:32:35.759084 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7hh2b\" (UniqueName: \"kubernetes.io/projected/fc654672-12df-4a71-827f-1951234b78e8-kube-api-access-7hh2b\") pod \"console-d6c6bcfbb-fjcg2\" (UID: \"fc654672-12df-4a71-827f-1951234b78e8\") " pod="openshift-console/console-d6c6bcfbb-fjcg2" Jul 09 15:32:35.878463 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:32:35.878372 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-d6c6bcfbb-fjcg2" Jul 09 15:32:36.011676 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:32:36.011644 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-d6c6bcfbb-fjcg2"] Jul 09 15:32:36.016255 ip-10-0-133-183 kubenswrapper[2573]: W0709 15:32:36.016230 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfc654672_12df_4a71_827f_1951234b78e8.slice/crio-b3e6304b05152a6668ceea7e5548ccaaf1cc2bf2e23088167def150941bcba9e WatchSource:0}: Error finding container b3e6304b05152a6668ceea7e5548ccaaf1cc2bf2e23088167def150941bcba9e: Status 404 returned error can't find the container with id b3e6304b05152a6668ceea7e5548ccaaf1cc2bf2e23088167def150941bcba9e Jul 09 15:32:36.172847 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:32:36.172754 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-d6c6bcfbb-fjcg2" event={"ID":"fc654672-12df-4a71-827f-1951234b78e8","Type":"ContainerStarted","Data":"b3e6304b05152a6668ceea7e5548ccaaf1cc2bf2e23088167def150941bcba9e"} Jul 09 15:32:36.210224 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:32:36.210190 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-ffqv6_b9e43e24-97dc-464d-a445-2100e1b546a9/dns/0.log" Jul 09 15:32:36.389494 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:32:36.389471 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-ffqv6_b9e43e24-97dc-464d-a445-2100e1b546a9/kube-rbac-proxy/0.log" Jul 09 15:32:36.748565 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:32:36.748483 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/8c2fc9e6-25b9-4d1d-9ea0-c6de5f1367e2-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-5kk64\" (UID: \"8c2fc9e6-25b9-4d1d-9ea0-c6de5f1367e2\") " pod="openshift-insights/insights-runtime-extractor-5kk64" Jul 09 15:32:36.748976 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:32:36.748568 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/8c2fc9e6-25b9-4d1d-9ea0-c6de5f1367e2-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-5kk64\" (UID: \"8c2fc9e6-25b9-4d1d-9ea0-c6de5f1367e2\") " pod="openshift-insights/insights-runtime-extractor-5kk64" Jul 09 15:32:36.748976 ip-10-0-133-183 kubenswrapper[2573]: E0709 15:32:36.748632 2573 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 09 15:32:36.748976 ip-10-0-133-183 kubenswrapper[2573]: E0709 15:32:36.748683 2573 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 09 15:32:36.748976 ip-10-0-133-183 kubenswrapper[2573]: E0709 15:32:36.748717 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8c2fc9e6-25b9-4d1d-9ea0-c6de5f1367e2-insights-runtime-extractor-tls podName:8c2fc9e6-25b9-4d1d-9ea0-c6de5f1367e2 nodeName:}" failed. No retries permitted until 2026-07-09 15:32:40.748694809 +0000 UTC m=+82.470201673 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/8c2fc9e6-25b9-4d1d-9ea0-c6de5f1367e2-insights-runtime-extractor-tls") pod "insights-runtime-extractor-5kk64" (UID: "8c2fc9e6-25b9-4d1d-9ea0-c6de5f1367e2") : secret "insights-runtime-extractor-tls" not found Jul 09 15:32:36.748976 ip-10-0-133-183 kubenswrapper[2573]: E0709 15:32:36.748743 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/8c2fc9e6-25b9-4d1d-9ea0-c6de5f1367e2-kube-rbac-proxy-cm podName:8c2fc9e6-25b9-4d1d-9ea0-c6de5f1367e2 nodeName:}" failed. No retries permitted until 2026-07-09 15:32:40.748724908 +0000 UTC m=+82.470231757 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/8c2fc9e6-25b9-4d1d-9ea0-c6de5f1367e2-kube-rbac-proxy-cm") pod "insights-runtime-extractor-5kk64" (UID: "8c2fc9e6-25b9-4d1d-9ea0-c6de5f1367e2") : configmap "kube-rbac-proxy" not found Jul 09 15:32:37.186511 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:32:37.186432 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-b5c8h_9ad54b5e-c82e-480c-814f-e1ab55bf32fb/dns-node-resolver/0.log" Jul 09 15:32:37.787204 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:32:37.787173 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_image-registry-7c768548b6-5tl98_81be5afd-69ba-4d11-95d0-efdb0f62bad0/registry/0.log" Jul 09 15:32:37.850749 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:32:37.850723 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-7c768548b6-5tl98" Jul 09 15:32:38.588206 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:32:38.588186 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-trnjd_1094a821-3b3e-46d8-ac90-64c7ae6e8cad/node-ca/0.log" Jul 09 15:32:39.185699 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:32:39.185665 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-d6c6bcfbb-fjcg2" event={"ID":"fc654672-12df-4a71-827f-1951234b78e8","Type":"ContainerStarted","Data":"30fdc0ef2a5ca8a9c520688d93554199622d08dd2f4b5196a572abd21154e67c"} Jul 09 15:32:39.198377 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:32:39.198333 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-fnfp8_d4b4e91f-6d12-45a0-a779-3fcda0e75f08/serve-healthcheck-canary/0.log" Jul 09 15:32:39.208844 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:32:39.208802 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-d6c6bcfbb-fjcg2" podStartSLOduration=1.6522166 podStartE2EDuration="4.208785827s" totalImagesPullingTime="2.556569227s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 15:32:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 15:32:39.208564227 +0000 UTC m=+80.930071097" watchObservedRunningTime="2026-07-09 15:32:39.208785827 +0000 UTC m=+80.930292696" Jul 09 15:32:40.780343 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:32:40.780209 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/8c2fc9e6-25b9-4d1d-9ea0-c6de5f1367e2-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-5kk64\" (UID: \"8c2fc9e6-25b9-4d1d-9ea0-c6de5f1367e2\") " pod="openshift-insights/insights-runtime-extractor-5kk64" Jul 09 15:32:40.780343 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:32:40.780272 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/8c2fc9e6-25b9-4d1d-9ea0-c6de5f1367e2-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-5kk64\" (UID: \"8c2fc9e6-25b9-4d1d-9ea0-c6de5f1367e2\") " pod="openshift-insights/insights-runtime-extractor-5kk64" Jul 09 15:32:40.780820 ip-10-0-133-183 kubenswrapper[2573]: E0709 15:32:40.780354 2573 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 09 15:32:40.780820 ip-10-0-133-183 kubenswrapper[2573]: E0709 15:32:40.780369 2573 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 09 15:32:40.780820 ip-10-0-133-183 kubenswrapper[2573]: E0709 15:32:40.780417 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/8c2fc9e6-25b9-4d1d-9ea0-c6de5f1367e2-kube-rbac-proxy-cm podName:8c2fc9e6-25b9-4d1d-9ea0-c6de5f1367e2 nodeName:}" failed. No retries permitted until 2026-07-09 15:32:48.780401844 +0000 UTC m=+90.501908691 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/8c2fc9e6-25b9-4d1d-9ea0-c6de5f1367e2-kube-rbac-proxy-cm") pod "insights-runtime-extractor-5kk64" (UID: "8c2fc9e6-25b9-4d1d-9ea0-c6de5f1367e2") : configmap "kube-rbac-proxy" not found Jul 09 15:32:40.780820 ip-10-0-133-183 kubenswrapper[2573]: E0709 15:32:40.780432 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8c2fc9e6-25b9-4d1d-9ea0-c6de5f1367e2-insights-runtime-extractor-tls podName:8c2fc9e6-25b9-4d1d-9ea0-c6de5f1367e2 nodeName:}" failed. No retries permitted until 2026-07-09 15:32:48.78042555 +0000 UTC m=+90.501932397 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/8c2fc9e6-25b9-4d1d-9ea0-c6de5f1367e2-insights-runtime-extractor-tls") pod "insights-runtime-extractor-5kk64" (UID: "8c2fc9e6-25b9-4d1d-9ea0-c6de5f1367e2") : secret "insights-runtime-extractor-tls" not found Jul 09 15:32:45.879075 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:32:45.879028 2573 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-d6c6bcfbb-fjcg2" Jul 09 15:32:45.879574 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:32:45.879102 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-d6c6bcfbb-fjcg2" Jul 09 15:32:45.883888 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:32:45.883846 2573 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-d6c6bcfbb-fjcg2" Jul 09 15:32:46.215730 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:32:46.215700 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-d6c6bcfbb-fjcg2" Jul 09 15:32:48.846204 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:32:48.846166 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/8c2fc9e6-25b9-4d1d-9ea0-c6de5f1367e2-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-5kk64\" (UID: \"8c2fc9e6-25b9-4d1d-9ea0-c6de5f1367e2\") " pod="openshift-insights/insights-runtime-extractor-5kk64" Jul 09 15:32:48.846582 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:32:48.846226 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/8c2fc9e6-25b9-4d1d-9ea0-c6de5f1367e2-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-5kk64\" (UID: \"8c2fc9e6-25b9-4d1d-9ea0-c6de5f1367e2\") " pod="openshift-insights/insights-runtime-extractor-5kk64" Jul 09 15:32:48.846818 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:32:48.846793 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/8c2fc9e6-25b9-4d1d-9ea0-c6de5f1367e2-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-5kk64\" (UID: \"8c2fc9e6-25b9-4d1d-9ea0-c6de5f1367e2\") " pod="openshift-insights/insights-runtime-extractor-5kk64" Jul 09 15:32:48.848631 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:32:48.848609 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/8c2fc9e6-25b9-4d1d-9ea0-c6de5f1367e2-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-5kk64\" (UID: \"8c2fc9e6-25b9-4d1d-9ea0-c6de5f1367e2\") " pod="openshift-insights/insights-runtime-extractor-5kk64" Jul 09 15:32:48.905008 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:32:48.904964 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-5kk64" Jul 09 15:32:49.039626 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:32:49.039594 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-5kk64"] Jul 09 15:32:49.042784 ip-10-0-133-183 kubenswrapper[2573]: W0709 15:32:49.042759 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8c2fc9e6_25b9_4d1d_9ea0_c6de5f1367e2.slice/crio-c1ad944633d479445d8ac889932b8c72c94b653c5beb07e003d9428fc58d9705 WatchSource:0}: Error finding container c1ad944633d479445d8ac889932b8c72c94b653c5beb07e003d9428fc58d9705: Status 404 returned error can't find the container with id c1ad944633d479445d8ac889932b8c72c94b653c5beb07e003d9428fc58d9705 Jul 09 15:32:49.221650 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:32:49.221615 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-5kk64" event={"ID":"8c2fc9e6-25b9-4d1d-9ea0-c6de5f1367e2","Type":"ContainerStarted","Data":"c1ad944633d479445d8ac889932b8c72c94b653c5beb07e003d9428fc58d9705"} Jul 09 15:32:52.871057 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:32:52.871001 2573 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-image-registry/image-registry-7c768548b6-5tl98" podUID="81be5afd-69ba-4d11-95d0-efdb0f62bad0" containerName="registry" containerID="cri-o://5e962fa8b5472577d1c52638b944f0aab609f29ef0212fb93138e09e80a02c92" gracePeriod=30 Jul 09 15:32:53.114306 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:32:53.114275 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-7c768548b6-5tl98" Jul 09 15:32:53.180821 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:32:53.180736 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/81be5afd-69ba-4d11-95d0-efdb0f62bad0-trusted-ca\") pod \"81be5afd-69ba-4d11-95d0-efdb0f62bad0\" (UID: \"81be5afd-69ba-4d11-95d0-efdb0f62bad0\") " Jul 09 15:32:53.180821 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:32:53.180778 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-q8mdg\" (UniqueName: \"kubernetes.io/projected/81be5afd-69ba-4d11-95d0-efdb0f62bad0-kube-api-access-q8mdg\") pod \"81be5afd-69ba-4d11-95d0-efdb0f62bad0\" (UID: \"81be5afd-69ba-4d11-95d0-efdb0f62bad0\") " Jul 09 15:32:53.180821 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:32:53.180822 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/81be5afd-69ba-4d11-95d0-efdb0f62bad0-registry-tls\") pod \"81be5afd-69ba-4d11-95d0-efdb0f62bad0\" (UID: \"81be5afd-69ba-4d11-95d0-efdb0f62bad0\") " Jul 09 15:32:53.181103 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:32:53.180842 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/81be5afd-69ba-4d11-95d0-efdb0f62bad0-ca-trust-extracted\") pod \"81be5afd-69ba-4d11-95d0-efdb0f62bad0\" (UID: \"81be5afd-69ba-4d11-95d0-efdb0f62bad0\") " Jul 09 15:32:53.181103 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:32:53.180896 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/81be5afd-69ba-4d11-95d0-efdb0f62bad0-image-registry-private-configuration\") pod \"81be5afd-69ba-4d11-95d0-efdb0f62bad0\" (UID: \"81be5afd-69ba-4d11-95d0-efdb0f62bad0\") " Jul 09 15:32:53.181103 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:32:53.180921 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/81be5afd-69ba-4d11-95d0-efdb0f62bad0-bound-sa-token\") pod \"81be5afd-69ba-4d11-95d0-efdb0f62bad0\" (UID: \"81be5afd-69ba-4d11-95d0-efdb0f62bad0\") " Jul 09 15:32:53.181103 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:32:53.180954 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/81be5afd-69ba-4d11-95d0-efdb0f62bad0-installation-pull-secrets\") pod \"81be5afd-69ba-4d11-95d0-efdb0f62bad0\" (UID: \"81be5afd-69ba-4d11-95d0-efdb0f62bad0\") " Jul 09 15:32:53.181103 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:32:53.180991 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/81be5afd-69ba-4d11-95d0-efdb0f62bad0-registry-certificates\") pod \"81be5afd-69ba-4d11-95d0-efdb0f62bad0\" (UID: \"81be5afd-69ba-4d11-95d0-efdb0f62bad0\") " Jul 09 15:32:53.181347 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:32:53.181240 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/81be5afd-69ba-4d11-95d0-efdb0f62bad0-trusted-ca" (OuterVolumeSpecName: "trusted-ca") pod "81be5afd-69ba-4d11-95d0-efdb0f62bad0" (UID: "81be5afd-69ba-4d11-95d0-efdb0f62bad0"). InnerVolumeSpecName "trusted-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 09 15:32:53.181665 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:32:53.181617 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/81be5afd-69ba-4d11-95d0-efdb0f62bad0-registry-certificates" (OuterVolumeSpecName: "registry-certificates") pod "81be5afd-69ba-4d11-95d0-efdb0f62bad0" (UID: "81be5afd-69ba-4d11-95d0-efdb0f62bad0"). InnerVolumeSpecName "registry-certificates". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 09 15:32:53.183536 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:32:53.183508 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/81be5afd-69ba-4d11-95d0-efdb0f62bad0-registry-tls" (OuterVolumeSpecName: "registry-tls") pod "81be5afd-69ba-4d11-95d0-efdb0f62bad0" (UID: "81be5afd-69ba-4d11-95d0-efdb0f62bad0"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 09 15:32:53.183665 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:32:53.183636 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/81be5afd-69ba-4d11-95d0-efdb0f62bad0-kube-api-access-q8mdg" (OuterVolumeSpecName: "kube-api-access-q8mdg") pod "81be5afd-69ba-4d11-95d0-efdb0f62bad0" (UID: "81be5afd-69ba-4d11-95d0-efdb0f62bad0"). InnerVolumeSpecName "kube-api-access-q8mdg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 09 15:32:53.183711 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:32:53.183681 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/81be5afd-69ba-4d11-95d0-efdb0f62bad0-bound-sa-token" (OuterVolumeSpecName: "bound-sa-token") pod "81be5afd-69ba-4d11-95d0-efdb0f62bad0" (UID: "81be5afd-69ba-4d11-95d0-efdb0f62bad0"). InnerVolumeSpecName "bound-sa-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 09 15:32:53.183761 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:32:53.183684 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/81be5afd-69ba-4d11-95d0-efdb0f62bad0-installation-pull-secrets" (OuterVolumeSpecName: "installation-pull-secrets") pod "81be5afd-69ba-4d11-95d0-efdb0f62bad0" (UID: "81be5afd-69ba-4d11-95d0-efdb0f62bad0"). InnerVolumeSpecName "installation-pull-secrets". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 09 15:32:53.183842 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:32:53.183816 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/81be5afd-69ba-4d11-95d0-efdb0f62bad0-image-registry-private-configuration" (OuterVolumeSpecName: "image-registry-private-configuration") pod "81be5afd-69ba-4d11-95d0-efdb0f62bad0" (UID: "81be5afd-69ba-4d11-95d0-efdb0f62bad0"). InnerVolumeSpecName "image-registry-private-configuration". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 09 15:32:53.189791 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:32:53.189768 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/81be5afd-69ba-4d11-95d0-efdb0f62bad0-ca-trust-extracted" (OuterVolumeSpecName: "ca-trust-extracted") pod "81be5afd-69ba-4d11-95d0-efdb0f62bad0" (UID: "81be5afd-69ba-4d11-95d0-efdb0f62bad0"). InnerVolumeSpecName "ca-trust-extracted". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 09 15:32:53.237038 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:32:53.237009 2573 generic.go:356] "Generic (PLEG): container finished" podID="81be5afd-69ba-4d11-95d0-efdb0f62bad0" containerID="5e962fa8b5472577d1c52638b944f0aab609f29ef0212fb93138e09e80a02c92" exitCode=0 Jul 09 15:32:53.237185 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:32:53.237052 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-7c768548b6-5tl98" Jul 09 15:32:53.237185 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:32:53.237102 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-7c768548b6-5tl98" event={"ID":"81be5afd-69ba-4d11-95d0-efdb0f62bad0","Type":"ContainerDied","Data":"5e962fa8b5472577d1c52638b944f0aab609f29ef0212fb93138e09e80a02c92"} Jul 09 15:32:53.237185 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:32:53.237159 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-7c768548b6-5tl98" event={"ID":"81be5afd-69ba-4d11-95d0-efdb0f62bad0","Type":"ContainerDied","Data":"ca80b44581823c2bc67be7fdb9bba88189543a7633d48280f66d43896c550b93"} Jul 09 15:32:53.237339 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:32:53.237201 2573 scope.go:117] "RemoveContainer" containerID="5e962fa8b5472577d1c52638b944f0aab609f29ef0212fb93138e09e80a02c92" Jul 09 15:32:53.238645 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:32:53.238619 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-5kk64" event={"ID":"8c2fc9e6-25b9-4d1d-9ea0-c6de5f1367e2","Type":"ContainerStarted","Data":"ffb741cc74eaa4652e6ecfc711c1e8657f1a4a3c0aaa83e8eb576f0db06380dd"} Jul 09 15:32:53.246300 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:32:53.246284 2573 scope.go:117] "RemoveContainer" containerID="5e962fa8b5472577d1c52638b944f0aab609f29ef0212fb93138e09e80a02c92" Jul 09 15:32:53.246581 ip-10-0-133-183 kubenswrapper[2573]: E0709 15:32:53.246563 2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5e962fa8b5472577d1c52638b944f0aab609f29ef0212fb93138e09e80a02c92\": container with ID starting with 5e962fa8b5472577d1c52638b944f0aab609f29ef0212fb93138e09e80a02c92 not found: ID does not exist" containerID="5e962fa8b5472577d1c52638b944f0aab609f29ef0212fb93138e09e80a02c92" Jul 09 15:32:53.246639 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:32:53.246590 2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5e962fa8b5472577d1c52638b944f0aab609f29ef0212fb93138e09e80a02c92"} err="failed to get container status \"5e962fa8b5472577d1c52638b944f0aab609f29ef0212fb93138e09e80a02c92\": rpc error: code = NotFound desc = could not find container \"5e962fa8b5472577d1c52638b944f0aab609f29ef0212fb93138e09e80a02c92\": container with ID starting with 5e962fa8b5472577d1c52638b944f0aab609f29ef0212fb93138e09e80a02c92 not found: ID does not exist" Jul 09 15:32:53.259328 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:32:53.259307 2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-7c768548b6-5tl98"] Jul 09 15:32:53.264364 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:32:53.264326 2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-image-registry/image-registry-7c768548b6-5tl98"] Jul 09 15:32:53.282474 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:32:53.282448 2573 reconciler_common.go:299] "Volume detached for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/81be5afd-69ba-4d11-95d0-efdb0f62bad0-registry-certificates\") on node \"ip-10-0-133-183.ec2.internal\" DevicePath \"\"" Jul 09 15:32:53.282474 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:32:53.282473 2573 reconciler_common.go:299] "Volume detached for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/81be5afd-69ba-4d11-95d0-efdb0f62bad0-trusted-ca\") on node \"ip-10-0-133-183.ec2.internal\" DevicePath \"\"" Jul 09 15:32:53.282599 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:32:53.282486 2573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q8mdg\" (UniqueName: \"kubernetes.io/projected/81be5afd-69ba-4d11-95d0-efdb0f62bad0-kube-api-access-q8mdg\") on node \"ip-10-0-133-183.ec2.internal\" DevicePath \"\"" Jul 09 15:32:53.282599 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:32:53.282498 2573 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/81be5afd-69ba-4d11-95d0-efdb0f62bad0-registry-tls\") on node \"ip-10-0-133-183.ec2.internal\" DevicePath \"\"" Jul 09 15:32:53.282599 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:32:53.282510 2573 reconciler_common.go:299] "Volume detached for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/81be5afd-69ba-4d11-95d0-efdb0f62bad0-ca-trust-extracted\") on node \"ip-10-0-133-183.ec2.internal\" DevicePath \"\"" Jul 09 15:32:53.282599 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:32:53.282523 2573 reconciler_common.go:299] "Volume detached for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/81be5afd-69ba-4d11-95d0-efdb0f62bad0-image-registry-private-configuration\") on node \"ip-10-0-133-183.ec2.internal\" DevicePath \"\"" Jul 09 15:32:53.282599 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:32:53.282537 2573 reconciler_common.go:299] "Volume detached for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/81be5afd-69ba-4d11-95d0-efdb0f62bad0-bound-sa-token\") on node \"ip-10-0-133-183.ec2.internal\" DevicePath \"\"" Jul 09 15:32:53.282599 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:32:53.282550 2573 reconciler_common.go:299] "Volume detached for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/81be5afd-69ba-4d11-95d0-efdb0f62bad0-installation-pull-secrets\") on node \"ip-10-0-133-183.ec2.internal\" DevicePath \"\"" Jul 09 15:32:54.246290 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:32:54.246247 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-5kk64" event={"ID":"8c2fc9e6-25b9-4d1d-9ea0-c6de5f1367e2","Type":"ContainerStarted","Data":"2574afa461c533585423f575e7f5ad3c72458684186108e93f20220175e7c314"} Jul 09 15:32:54.809506 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:32:54.809473 2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="81be5afd-69ba-4d11-95d0-efdb0f62bad0" path="/var/lib/kubelet/pods/81be5afd-69ba-4d11-95d0-efdb0f62bad0/volumes" Jul 09 15:32:59.479144 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:32:59.479108 2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-64785b6675-smrzt"] Jul 09 15:32:59.479621 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:32:59.479428 2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="81be5afd-69ba-4d11-95d0-efdb0f62bad0" containerName="registry" Jul 09 15:32:59.479621 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:32:59.479445 2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="81be5afd-69ba-4d11-95d0-efdb0f62bad0" containerName="registry" Jul 09 15:32:59.479621 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:32:59.479491 2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="81be5afd-69ba-4d11-95d0-efdb0f62bad0" containerName="registry" Jul 09 15:32:59.485963 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:32:59.485939 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-64785b6675-smrzt" Jul 09 15:32:59.489057 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:32:59.489034 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-tls\"" Jul 09 15:32:59.489223 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:32:59.489084 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\"" Jul 09 15:32:59.490175 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:32:59.490104 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\"" Jul 09 15:32:59.490175 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:32:59.490151 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\"" Jul 09 15:32:59.490448 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:32:59.490174 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-dockercfg-4njlm\"" Jul 09 15:32:59.490524 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:32:59.490479 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-kube-rbac-proxy-config\"" Jul 09 15:32:59.492687 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:32:59.492665 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-64785b6675-smrzt"] Jul 09 15:32:59.629643 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:32:59.629606 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gdvkn\" (UniqueName: \"kubernetes.io/projected/b7e1c1ed-4b4d-4062-b4d2-d9f3759d4756-kube-api-access-gdvkn\") pod \"prometheus-operator-64785b6675-smrzt\" (UID: \"b7e1c1ed-4b4d-4062-b4d2-d9f3759d4756\") " pod="openshift-monitoring/prometheus-operator-64785b6675-smrzt" Jul 09 15:32:59.629843 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:32:59.629654 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/b7e1c1ed-4b4d-4062-b4d2-d9f3759d4756-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-64785b6675-smrzt\" (UID: \"b7e1c1ed-4b4d-4062-b4d2-d9f3759d4756\") " pod="openshift-monitoring/prometheus-operator-64785b6675-smrzt" Jul 09 15:32:59.629843 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:32:59.629784 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/b7e1c1ed-4b4d-4062-b4d2-d9f3759d4756-metrics-client-ca\") pod \"prometheus-operator-64785b6675-smrzt\" (UID: \"b7e1c1ed-4b4d-4062-b4d2-d9f3759d4756\") " pod="openshift-monitoring/prometheus-operator-64785b6675-smrzt" Jul 09 15:32:59.629843 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:32:59.629837 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/b7e1c1ed-4b4d-4062-b4d2-d9f3759d4756-prometheus-operator-tls\") pod \"prometheus-operator-64785b6675-smrzt\" (UID: \"b7e1c1ed-4b4d-4062-b4d2-d9f3759d4756\") " pod="openshift-monitoring/prometheus-operator-64785b6675-smrzt" Jul 09 15:32:59.731210 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:32:59.731113 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/b7e1c1ed-4b4d-4062-b4d2-d9f3759d4756-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-64785b6675-smrzt\" (UID: \"b7e1c1ed-4b4d-4062-b4d2-d9f3759d4756\") " pod="openshift-monitoring/prometheus-operator-64785b6675-smrzt" Jul 09 15:32:59.731363 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:32:59.731221 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/b7e1c1ed-4b4d-4062-b4d2-d9f3759d4756-metrics-client-ca\") pod \"prometheus-operator-64785b6675-smrzt\" (UID: \"b7e1c1ed-4b4d-4062-b4d2-d9f3759d4756\") " pod="openshift-monitoring/prometheus-operator-64785b6675-smrzt" Jul 09 15:32:59.731363 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:32:59.731267 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/b7e1c1ed-4b4d-4062-b4d2-d9f3759d4756-prometheus-operator-tls\") pod \"prometheus-operator-64785b6675-smrzt\" (UID: \"b7e1c1ed-4b4d-4062-b4d2-d9f3759d4756\") " pod="openshift-monitoring/prometheus-operator-64785b6675-smrzt" Jul 09 15:32:59.731363 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:32:59.731327 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gdvkn\" (UniqueName: \"kubernetes.io/projected/b7e1c1ed-4b4d-4062-b4d2-d9f3759d4756-kube-api-access-gdvkn\") pod \"prometheus-operator-64785b6675-smrzt\" (UID: \"b7e1c1ed-4b4d-4062-b4d2-d9f3759d4756\") " pod="openshift-monitoring/prometheus-operator-64785b6675-smrzt" Jul 09 15:32:59.731520 ip-10-0-133-183 kubenswrapper[2573]: E0709 15:32:59.731431 2573 secret.go:189] Couldn't get secret openshift-monitoring/prometheus-operator-tls: secret "prometheus-operator-tls" not found Jul 09 15:32:59.731520 ip-10-0-133-183 kubenswrapper[2573]: E0709 15:32:59.731509 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b7e1c1ed-4b4d-4062-b4d2-d9f3759d4756-prometheus-operator-tls podName:b7e1c1ed-4b4d-4062-b4d2-d9f3759d4756 nodeName:}" failed. No retries permitted until 2026-07-09 15:33:00.231486679 +0000 UTC m=+101.952993544 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "prometheus-operator-tls" (UniqueName: "kubernetes.io/secret/b7e1c1ed-4b4d-4062-b4d2-d9f3759d4756-prometheus-operator-tls") pod "prometheus-operator-64785b6675-smrzt" (UID: "b7e1c1ed-4b4d-4062-b4d2-d9f3759d4756") : secret "prometheus-operator-tls" not found Jul 09 15:32:59.732087 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:32:59.732063 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/b7e1c1ed-4b4d-4062-b4d2-d9f3759d4756-metrics-client-ca\") pod \"prometheus-operator-64785b6675-smrzt\" (UID: \"b7e1c1ed-4b4d-4062-b4d2-d9f3759d4756\") " pod="openshift-monitoring/prometheus-operator-64785b6675-smrzt" Jul 09 15:32:59.733926 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:32:59.733901 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/b7e1c1ed-4b4d-4062-b4d2-d9f3759d4756-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-64785b6675-smrzt\" (UID: \"b7e1c1ed-4b4d-4062-b4d2-d9f3759d4756\") " pod="openshift-monitoring/prometheus-operator-64785b6675-smrzt" Jul 09 15:32:59.742209 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:32:59.742184 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gdvkn\" (UniqueName: \"kubernetes.io/projected/b7e1c1ed-4b4d-4062-b4d2-d9f3759d4756-kube-api-access-gdvkn\") pod \"prometheus-operator-64785b6675-smrzt\" (UID: \"b7e1c1ed-4b4d-4062-b4d2-d9f3759d4756\") " pod="openshift-monitoring/prometheus-operator-64785b6675-smrzt" Jul 09 15:33:00.234398 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:33:00.234377 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/b7e1c1ed-4b4d-4062-b4d2-d9f3759d4756-prometheus-operator-tls\") pod \"prometheus-operator-64785b6675-smrzt\" (UID: \"b7e1c1ed-4b4d-4062-b4d2-d9f3759d4756\") " pod="openshift-monitoring/prometheus-operator-64785b6675-smrzt" Jul 09 15:33:00.236785 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:33:00.236763 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/b7e1c1ed-4b4d-4062-b4d2-d9f3759d4756-prometheus-operator-tls\") pod \"prometheus-operator-64785b6675-smrzt\" (UID: \"b7e1c1ed-4b4d-4062-b4d2-d9f3759d4756\") " pod="openshift-monitoring/prometheus-operator-64785b6675-smrzt" Jul 09 15:33:00.398674 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:33:00.398583 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-64785b6675-smrzt" Jul 09 15:33:00.549526 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:33:00.549498 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-64785b6675-smrzt"] Jul 09 15:33:00.552827 ip-10-0-133-183 kubenswrapper[2573]: W0709 15:33:00.552804 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb7e1c1ed_4b4d_4062_b4d2_d9f3759d4756.slice/crio-d268d0473df84096e64946900c193b7678c1d61c929b846f41d7fb99a58009f6 WatchSource:0}: Error finding container d268d0473df84096e64946900c193b7678c1d61c929b846f41d7fb99a58009f6: Status 404 returned error can't find the container with id d268d0473df84096e64946900c193b7678c1d61c929b846f41d7fb99a58009f6 Jul 09 15:33:01.157098 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:33:01.157066 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-sdbbl" Jul 09 15:33:01.273898 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:33:01.273844 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-5kk64" event={"ID":"8c2fc9e6-25b9-4d1d-9ea0-c6de5f1367e2","Type":"ContainerStarted","Data":"e964d35604ad6267f1d80ef86196823983a8b2f653c5cc41c62d0b7e48c386f4"} Jul 09 15:33:01.275175 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:33:01.275149 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-64785b6675-smrzt" event={"ID":"b7e1c1ed-4b4d-4062-b4d2-d9f3759d4756","Type":"ContainerStarted","Data":"d268d0473df84096e64946900c193b7678c1d61c929b846f41d7fb99a58009f6"} Jul 09 15:33:01.296370 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:33:01.296323 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-5kk64" podStartSLOduration=18.22915374 podStartE2EDuration="29.296306597s" totalImagesPullingTime="11.067152857s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-09 15:32:32 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 15:33:01.294681099 +0000 UTC m=+103.016187982" watchObservedRunningTime="2026-07-09 15:33:01.296306597 +0000 UTC m=+103.017813512" Jul 09 15:33:02.281972 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:33:02.281917 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-64785b6675-smrzt" event={"ID":"b7e1c1ed-4b4d-4062-b4d2-d9f3759d4756","Type":"ContainerStarted","Data":"b5f0233dc1407589fd6d613c1fca8e856c28f95b22f3efec6f0d7169fb93e008"} Jul 09 15:33:02.282418 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:33:02.282014 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-64785b6675-smrzt" event={"ID":"b7e1c1ed-4b4d-4062-b4d2-d9f3759d4756","Type":"ContainerStarted","Data":"900bdbf75a271146d8346572e47b71e68d3af365489e5f4aab1e7fbef6c37bf7"} Jul 09 15:33:02.302155 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:33:02.302103 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-64785b6675-smrzt" podStartSLOduration=1.741057349 podStartE2EDuration="3.302084702s" totalImagesPullingTime="1.561027353s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 15:32:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 15:33:02.300195144 +0000 UTC m=+104.021702016" watchObservedRunningTime="2026-07-09 15:33:02.302084702 +0000 UTC m=+104.023591574" Jul 09 15:33:03.937823 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:33:03.937785 2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/openshift-state-metrics-84bd68b896-q9km7"] Jul 09 15:33:03.970612 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:33:03.970582 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-84bd68b896-q9km7"] Jul 09 15:33:03.970778 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:33:03.970704 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-84bd68b896-q9km7" Jul 09 15:33:03.978741 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:33:03.978715 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-kube-rbac-proxy-config\"" Jul 09 15:33:03.978901 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:33:03.978829 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-tls\"" Jul 09 15:33:03.980099 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:33:03.980080 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-dockercfg-mdk6h\"" Jul 09 15:33:03.980279 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:33:03.980262 2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-4jbr9"] Jul 09 15:33:04.001814 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:33:04.001783 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-4jbr9" Jul 09 15:33:04.005345 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:33:04.005315 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\"" Jul 09 15:33:04.005471 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:33:04.005415 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-tffth\"" Jul 09 15:33:04.005692 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:33:04.005672 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\"" Jul 09 15:33:04.006654 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:33:04.006638 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\"" Jul 09 15:33:04.065984 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:33:04.065950 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/9889ca25-29ca-4226-8c5a-4e2eb8040868-metrics-client-ca\") pod \"openshift-state-metrics-84bd68b896-q9km7\" (UID: \"9889ca25-29ca-4226-8c5a-4e2eb8040868\") " pod="openshift-monitoring/openshift-state-metrics-84bd68b896-q9km7" Jul 09 15:33:04.066157 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:33:04.066002 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/9889ca25-29ca-4226-8c5a-4e2eb8040868-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-84bd68b896-q9km7\" (UID: \"9889ca25-29ca-4226-8c5a-4e2eb8040868\") " pod="openshift-monitoring/openshift-state-metrics-84bd68b896-q9km7" Jul 09 15:33:04.066157 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:33:04.066034 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-phb9b\" (UniqueName: \"kubernetes.io/projected/9889ca25-29ca-4226-8c5a-4e2eb8040868-kube-api-access-phb9b\") pod \"openshift-state-metrics-84bd68b896-q9km7\" (UID: \"9889ca25-29ca-4226-8c5a-4e2eb8040868\") " pod="openshift-monitoring/openshift-state-metrics-84bd68b896-q9km7" Jul 09 15:33:04.066157 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:33:04.066061 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/9889ca25-29ca-4226-8c5a-4e2eb8040868-openshift-state-metrics-tls\") pod \"openshift-state-metrics-84bd68b896-q9km7\" (UID: \"9889ca25-29ca-4226-8c5a-4e2eb8040868\") " pod="openshift-monitoring/openshift-state-metrics-84bd68b896-q9km7" Jul 09 15:33:04.166849 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:33:04.166810 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/9889ca25-29ca-4226-8c5a-4e2eb8040868-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-84bd68b896-q9km7\" (UID: \"9889ca25-29ca-4226-8c5a-4e2eb8040868\") " pod="openshift-monitoring/openshift-state-metrics-84bd68b896-q9km7" Jul 09 15:33:04.166849 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:33:04.166854 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4gvsx\" (UniqueName: \"kubernetes.io/projected/1886ffd8-a444-41ce-803c-58c58160faff-kube-api-access-4gvsx\") pod \"node-exporter-4jbr9\" (UID: \"1886ffd8-a444-41ce-803c-58c58160faff\") " pod="openshift-monitoring/node-exporter-4jbr9" Jul 09 15:33:04.167079 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:33:04.166904 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-phb9b\" (UniqueName: \"kubernetes.io/projected/9889ca25-29ca-4226-8c5a-4e2eb8040868-kube-api-access-phb9b\") pod \"openshift-state-metrics-84bd68b896-q9km7\" (UID: \"9889ca25-29ca-4226-8c5a-4e2eb8040868\") " pod="openshift-monitoring/openshift-state-metrics-84bd68b896-q9km7" Jul 09 15:33:04.167079 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:33:04.167046 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/9889ca25-29ca-4226-8c5a-4e2eb8040868-openshift-state-metrics-tls\") pod \"openshift-state-metrics-84bd68b896-q9km7\" (UID: \"9889ca25-29ca-4226-8c5a-4e2eb8040868\") " pod="openshift-monitoring/openshift-state-metrics-84bd68b896-q9km7" Jul 09 15:33:04.167455 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:33:04.167083 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/1886ffd8-a444-41ce-803c-58c58160faff-node-exporter-tls\") pod \"node-exporter-4jbr9\" (UID: \"1886ffd8-a444-41ce-803c-58c58160faff\") " pod="openshift-monitoring/node-exporter-4jbr9" Jul 09 15:33:04.167455 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:33:04.167129 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/1886ffd8-a444-41ce-803c-58c58160faff-node-exporter-wtmp\") pod \"node-exporter-4jbr9\" (UID: \"1886ffd8-a444-41ce-803c-58c58160faff\") " pod="openshift-monitoring/node-exporter-4jbr9" Jul 09 15:33:04.167455 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:33:04.167185 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/1886ffd8-a444-41ce-803c-58c58160faff-node-exporter-textfile\") pod \"node-exporter-4jbr9\" (UID: \"1886ffd8-a444-41ce-803c-58c58160faff\") " pod="openshift-monitoring/node-exporter-4jbr9" Jul 09 15:33:04.167455 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:33:04.167210 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/9889ca25-29ca-4226-8c5a-4e2eb8040868-metrics-client-ca\") pod \"openshift-state-metrics-84bd68b896-q9km7\" (UID: \"9889ca25-29ca-4226-8c5a-4e2eb8040868\") " pod="openshift-monitoring/openshift-state-metrics-84bd68b896-q9km7" Jul 09 15:33:04.167455 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:33:04.167236 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/1886ffd8-a444-41ce-803c-58c58160faff-root\") pod \"node-exporter-4jbr9\" (UID: \"1886ffd8-a444-41ce-803c-58c58160faff\") " pod="openshift-monitoring/node-exporter-4jbr9" Jul 09 15:33:04.167455 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:33:04.167260 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/1886ffd8-a444-41ce-803c-58c58160faff-node-exporter-accelerators-collector-config\") pod \"node-exporter-4jbr9\" (UID: \"1886ffd8-a444-41ce-803c-58c58160faff\") " pod="openshift-monitoring/node-exporter-4jbr9" Jul 09 15:33:04.167455 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:33:04.167298 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/1886ffd8-a444-41ce-803c-58c58160faff-sys\") pod \"node-exporter-4jbr9\" (UID: \"1886ffd8-a444-41ce-803c-58c58160faff\") " pod="openshift-monitoring/node-exporter-4jbr9" Jul 09 15:33:04.167455 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:33:04.167317 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/1886ffd8-a444-41ce-803c-58c58160faff-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-4jbr9\" (UID: \"1886ffd8-a444-41ce-803c-58c58160faff\") " pod="openshift-monitoring/node-exporter-4jbr9" Jul 09 15:33:04.167455 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:33:04.167339 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/1886ffd8-a444-41ce-803c-58c58160faff-metrics-client-ca\") pod \"node-exporter-4jbr9\" (UID: \"1886ffd8-a444-41ce-803c-58c58160faff\") " pod="openshift-monitoring/node-exporter-4jbr9" Jul 09 15:33:04.167864 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:33:04.167846 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/9889ca25-29ca-4226-8c5a-4e2eb8040868-metrics-client-ca\") pod \"openshift-state-metrics-84bd68b896-q9km7\" (UID: \"9889ca25-29ca-4226-8c5a-4e2eb8040868\") " pod="openshift-monitoring/openshift-state-metrics-84bd68b896-q9km7" Jul 09 15:33:04.169435 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:33:04.169409 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/9889ca25-29ca-4226-8c5a-4e2eb8040868-openshift-state-metrics-tls\") pod \"openshift-state-metrics-84bd68b896-q9km7\" (UID: \"9889ca25-29ca-4226-8c5a-4e2eb8040868\") " pod="openshift-monitoring/openshift-state-metrics-84bd68b896-q9km7" Jul 09 15:33:04.169527 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:33:04.169417 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/9889ca25-29ca-4226-8c5a-4e2eb8040868-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-84bd68b896-q9km7\" (UID: \"9889ca25-29ca-4226-8c5a-4e2eb8040868\") " pod="openshift-monitoring/openshift-state-metrics-84bd68b896-q9km7" Jul 09 15:33:04.181950 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:33:04.181923 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-phb9b\" (UniqueName: \"kubernetes.io/projected/9889ca25-29ca-4226-8c5a-4e2eb8040868-kube-api-access-phb9b\") pod \"openshift-state-metrics-84bd68b896-q9km7\" (UID: \"9889ca25-29ca-4226-8c5a-4e2eb8040868\") " pod="openshift-monitoring/openshift-state-metrics-84bd68b896-q9km7" Jul 09 15:33:04.267973 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:33:04.267935 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/1886ffd8-a444-41ce-803c-58c58160faff-root\") pod \"node-exporter-4jbr9\" (UID: \"1886ffd8-a444-41ce-803c-58c58160faff\") " pod="openshift-monitoring/node-exporter-4jbr9" Jul 09 15:33:04.267973 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:33:04.267978 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/1886ffd8-a444-41ce-803c-58c58160faff-node-exporter-accelerators-collector-config\") pod \"node-exporter-4jbr9\" (UID: \"1886ffd8-a444-41ce-803c-58c58160faff\") " pod="openshift-monitoring/node-exporter-4jbr9" Jul 09 15:33:04.268197 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:33:04.268001 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/1886ffd8-a444-41ce-803c-58c58160faff-sys\") pod \"node-exporter-4jbr9\" (UID: \"1886ffd8-a444-41ce-803c-58c58160faff\") " pod="openshift-monitoring/node-exporter-4jbr9" Jul 09 15:33:04.268197 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:33:04.268032 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/1886ffd8-a444-41ce-803c-58c58160faff-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-4jbr9\" (UID: \"1886ffd8-a444-41ce-803c-58c58160faff\") " pod="openshift-monitoring/node-exporter-4jbr9" Jul 09 15:33:04.268197 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:33:04.268051 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/1886ffd8-a444-41ce-803c-58c58160faff-root\") pod \"node-exporter-4jbr9\" (UID: \"1886ffd8-a444-41ce-803c-58c58160faff\") " pod="openshift-monitoring/node-exporter-4jbr9" Jul 09 15:33:04.268197 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:33:04.268078 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/1886ffd8-a444-41ce-803c-58c58160faff-sys\") pod \"node-exporter-4jbr9\" (UID: \"1886ffd8-a444-41ce-803c-58c58160faff\") " pod="openshift-monitoring/node-exporter-4jbr9" Jul 09 15:33:04.268197 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:33:04.268064 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/1886ffd8-a444-41ce-803c-58c58160faff-metrics-client-ca\") pod \"node-exporter-4jbr9\" (UID: \"1886ffd8-a444-41ce-803c-58c58160faff\") " pod="openshift-monitoring/node-exporter-4jbr9" Jul 09 15:33:04.268197 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:33:04.268149 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4gvsx\" (UniqueName: \"kubernetes.io/projected/1886ffd8-a444-41ce-803c-58c58160faff-kube-api-access-4gvsx\") pod \"node-exporter-4jbr9\" (UID: \"1886ffd8-a444-41ce-803c-58c58160faff\") " pod="openshift-monitoring/node-exporter-4jbr9" Jul 09 15:33:04.268197 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:33:04.268180 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/1886ffd8-a444-41ce-803c-58c58160faff-node-exporter-tls\") pod \"node-exporter-4jbr9\" (UID: \"1886ffd8-a444-41ce-803c-58c58160faff\") " pod="openshift-monitoring/node-exporter-4jbr9" Jul 09 15:33:04.268513 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:33:04.268215 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/1886ffd8-a444-41ce-803c-58c58160faff-node-exporter-wtmp\") pod \"node-exporter-4jbr9\" (UID: \"1886ffd8-a444-41ce-803c-58c58160faff\") " pod="openshift-monitoring/node-exporter-4jbr9" Jul 09 15:33:04.268513 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:33:04.268252 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/1886ffd8-a444-41ce-803c-58c58160faff-node-exporter-textfile\") pod \"node-exporter-4jbr9\" (UID: \"1886ffd8-a444-41ce-803c-58c58160faff\") " pod="openshift-monitoring/node-exporter-4jbr9" Jul 09 15:33:04.268513 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:33:04.268386 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/1886ffd8-a444-41ce-803c-58c58160faff-node-exporter-wtmp\") pod \"node-exporter-4jbr9\" (UID: \"1886ffd8-a444-41ce-803c-58c58160faff\") " pod="openshift-monitoring/node-exporter-4jbr9" Jul 09 15:33:04.268669 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:33:04.268643 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/1886ffd8-a444-41ce-803c-58c58160faff-node-exporter-textfile\") pod \"node-exporter-4jbr9\" (UID: \"1886ffd8-a444-41ce-803c-58c58160faff\") " pod="openshift-monitoring/node-exporter-4jbr9" Jul 09 15:33:04.268720 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:33:04.268701 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/1886ffd8-a444-41ce-803c-58c58160faff-metrics-client-ca\") pod \"node-exporter-4jbr9\" (UID: \"1886ffd8-a444-41ce-803c-58c58160faff\") " pod="openshift-monitoring/node-exporter-4jbr9" Jul 09 15:33:04.268777 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:33:04.268720 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/1886ffd8-a444-41ce-803c-58c58160faff-node-exporter-accelerators-collector-config\") pod \"node-exporter-4jbr9\" (UID: \"1886ffd8-a444-41ce-803c-58c58160faff\") " pod="openshift-monitoring/node-exporter-4jbr9" Jul 09 15:33:04.270478 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:33:04.270455 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/1886ffd8-a444-41ce-803c-58c58160faff-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-4jbr9\" (UID: \"1886ffd8-a444-41ce-803c-58c58160faff\") " pod="openshift-monitoring/node-exporter-4jbr9" Jul 09 15:33:04.270569 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:33:04.270491 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/1886ffd8-a444-41ce-803c-58c58160faff-node-exporter-tls\") pod \"node-exporter-4jbr9\" (UID: \"1886ffd8-a444-41ce-803c-58c58160faff\") " pod="openshift-monitoring/node-exporter-4jbr9" Jul 09 15:33:04.279362 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:33:04.279346 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-84bd68b896-q9km7" Jul 09 15:33:04.279960 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:33:04.279943 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4gvsx\" (UniqueName: \"kubernetes.io/projected/1886ffd8-a444-41ce-803c-58c58160faff-kube-api-access-4gvsx\") pod \"node-exporter-4jbr9\" (UID: \"1886ffd8-a444-41ce-803c-58c58160faff\") " pod="openshift-monitoring/node-exporter-4jbr9" Jul 09 15:33:04.311261 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:33:04.311225 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-4jbr9" Jul 09 15:33:04.345129 ip-10-0-133-183 kubenswrapper[2573]: W0709 15:33:04.345102 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1886ffd8_a444_41ce_803c_58c58160faff.slice/crio-a2f7b67b5e5044200b215e67221a347ac435a202b4f6f54b319cef78543cb15a WatchSource:0}: Error finding container a2f7b67b5e5044200b215e67221a347ac435a202b4f6f54b319cef78543cb15a: Status 404 returned error can't find the container with id a2f7b67b5e5044200b215e67221a347ac435a202b4f6f54b319cef78543cb15a Jul 09 15:33:04.474004 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:33:04.473976 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-84bd68b896-q9km7"] Jul 09 15:33:04.476280 ip-10-0-133-183 kubenswrapper[2573]: W0709 15:33:04.476252 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9889ca25_29ca_4226_8c5a_4e2eb8040868.slice/crio-35a0b5e34025d092f7ede64953eb7ed2cba1aa35aed8165ecff26475365611d4 WatchSource:0}: Error finding container 35a0b5e34025d092f7ede64953eb7ed2cba1aa35aed8165ecff26475365611d4: Status 404 returned error can't find the container with id 35a0b5e34025d092f7ede64953eb7ed2cba1aa35aed8165ecff26475365611d4 Jul 09 15:33:05.215713 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:33:05.215674 2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Jul 09 15:33:05.229206 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:33:05.229172 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Jul 09 15:33:05.239869 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:33:05.239841 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\"" Jul 09 15:33:05.240009 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:33:05.239848 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\"" Jul 09 15:33:05.240009 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:33:05.239904 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\"" Jul 09 15:33:05.240009 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:33:05.239929 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\"" Jul 09 15:33:05.241719 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:33:05.241583 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-zvbwt\"" Jul 09 15:33:05.243185 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:33:05.243162 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\"" Jul 09 15:33:05.243303 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:33:05.243199 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\"" Jul 09 15:33:05.243363 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:33:05.243353 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\"" Jul 09 15:33:05.243527 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:33:05.243509 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\"" Jul 09 15:33:05.245040 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:33:05.245019 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\"" Jul 09 15:33:05.245573 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:33:05.245555 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Jul 09 15:33:05.293309 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:33:05.293274 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-4jbr9" event={"ID":"1886ffd8-a444-41ce-803c-58c58160faff","Type":"ContainerStarted","Data":"a2f7b67b5e5044200b215e67221a347ac435a202b4f6f54b319cef78543cb15a"} Jul 09 15:33:05.295120 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:33:05.295094 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-84bd68b896-q9km7" event={"ID":"9889ca25-29ca-4226-8c5a-4e2eb8040868","Type":"ContainerStarted","Data":"0222fde1f80e0d9a0bb59296ae318715f2a49b09dcb9e6ba20a3701169d7c7ee"} Jul 09 15:33:05.295244 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:33:05.295146 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-84bd68b896-q9km7" event={"ID":"9889ca25-29ca-4226-8c5a-4e2eb8040868","Type":"ContainerStarted","Data":"092df12bb48d35be110fbabfddc7e3d0d312f5cbda51fe1bc1ca6dd6cfb8de1f"} Jul 09 15:33:05.295244 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:33:05.295166 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-84bd68b896-q9km7" event={"ID":"9889ca25-29ca-4226-8c5a-4e2eb8040868","Type":"ContainerStarted","Data":"35a0b5e34025d092f7ede64953eb7ed2cba1aa35aed8165ecff26475365611d4"} Jul 09 15:33:05.380686 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:33:05.380643 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/d060b445-2d60-481c-a52d-0fd1c8025564-config-volume\") pod \"alertmanager-main-0\" (UID: \"d060b445-2d60-481c-a52d-0fd1c8025564\") " pod="openshift-monitoring/alertmanager-main-0" Jul 09 15:33:05.380686 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:33:05.380685 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tjhrd\" (UniqueName: \"kubernetes.io/projected/d060b445-2d60-481c-a52d-0fd1c8025564-kube-api-access-tjhrd\") pod \"alertmanager-main-0\" (UID: \"d060b445-2d60-481c-a52d-0fd1c8025564\") " pod="openshift-monitoring/alertmanager-main-0" Jul 09 15:33:05.381005 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:33:05.380709 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/d060b445-2d60-481c-a52d-0fd1c8025564-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"d060b445-2d60-481c-a52d-0fd1c8025564\") " pod="openshift-monitoring/alertmanager-main-0" Jul 09 15:33:05.381005 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:33:05.380810 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/d060b445-2d60-481c-a52d-0fd1c8025564-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"d060b445-2d60-481c-a52d-0fd1c8025564\") " pod="openshift-monitoring/alertmanager-main-0" Jul 09 15:33:05.381005 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:33:05.380844 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d060b445-2d60-481c-a52d-0fd1c8025564-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"d060b445-2d60-481c-a52d-0fd1c8025564\") " pod="openshift-monitoring/alertmanager-main-0" Jul 09 15:33:05.381005 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:33:05.380909 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/d060b445-2d60-481c-a52d-0fd1c8025564-web-config\") pod \"alertmanager-main-0\" (UID: \"d060b445-2d60-481c-a52d-0fd1c8025564\") " pod="openshift-monitoring/alertmanager-main-0" Jul 09 15:33:05.381005 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:33:05.380962 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/d060b445-2d60-481c-a52d-0fd1c8025564-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"d060b445-2d60-481c-a52d-0fd1c8025564\") " pod="openshift-monitoring/alertmanager-main-0" Jul 09 15:33:05.381005 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:33:05.381002 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/d060b445-2d60-481c-a52d-0fd1c8025564-tls-assets\") pod \"alertmanager-main-0\" (UID: \"d060b445-2d60-481c-a52d-0fd1c8025564\") " pod="openshift-monitoring/alertmanager-main-0" Jul 09 15:33:05.381291 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:33:05.381029 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/d060b445-2d60-481c-a52d-0fd1c8025564-config-out\") pod \"alertmanager-main-0\" (UID: \"d060b445-2d60-481c-a52d-0fd1c8025564\") " pod="openshift-monitoring/alertmanager-main-0" Jul 09 15:33:05.381291 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:33:05.381060 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/d060b445-2d60-481c-a52d-0fd1c8025564-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"d060b445-2d60-481c-a52d-0fd1c8025564\") " pod="openshift-monitoring/alertmanager-main-0" Jul 09 15:33:05.381291 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:33:05.381086 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/d060b445-2d60-481c-a52d-0fd1c8025564-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"d060b445-2d60-481c-a52d-0fd1c8025564\") " pod="openshift-monitoring/alertmanager-main-0" Jul 09 15:33:05.381291 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:33:05.381116 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/d060b445-2d60-481c-a52d-0fd1c8025564-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"d060b445-2d60-481c-a52d-0fd1c8025564\") " pod="openshift-monitoring/alertmanager-main-0" Jul 09 15:33:05.381291 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:33:05.381145 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/d060b445-2d60-481c-a52d-0fd1c8025564-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"d060b445-2d60-481c-a52d-0fd1c8025564\") " pod="openshift-monitoring/alertmanager-main-0" Jul 09 15:33:05.482116 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:33:05.482022 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/d060b445-2d60-481c-a52d-0fd1c8025564-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"d060b445-2d60-481c-a52d-0fd1c8025564\") " pod="openshift-monitoring/alertmanager-main-0" Jul 09 15:33:05.482116 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:33:05.482068 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/d060b445-2d60-481c-a52d-0fd1c8025564-config-volume\") pod \"alertmanager-main-0\" (UID: \"d060b445-2d60-481c-a52d-0fd1c8025564\") " pod="openshift-monitoring/alertmanager-main-0" Jul 09 15:33:05.482116 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:33:05.482084 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tjhrd\" (UniqueName: \"kubernetes.io/projected/d060b445-2d60-481c-a52d-0fd1c8025564-kube-api-access-tjhrd\") pod \"alertmanager-main-0\" (UID: \"d060b445-2d60-481c-a52d-0fd1c8025564\") " pod="openshift-monitoring/alertmanager-main-0" Jul 09 15:33:05.482459 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:33:05.482120 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/d060b445-2d60-481c-a52d-0fd1c8025564-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"d060b445-2d60-481c-a52d-0fd1c8025564\") " pod="openshift-monitoring/alertmanager-main-0" Jul 09 15:33:05.482459 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:33:05.482167 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/d060b445-2d60-481c-a52d-0fd1c8025564-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"d060b445-2d60-481c-a52d-0fd1c8025564\") " pod="openshift-monitoring/alertmanager-main-0" Jul 09 15:33:05.482459 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:33:05.482191 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d060b445-2d60-481c-a52d-0fd1c8025564-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"d060b445-2d60-481c-a52d-0fd1c8025564\") " pod="openshift-monitoring/alertmanager-main-0" Jul 09 15:33:05.482459 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:33:05.482216 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/d060b445-2d60-481c-a52d-0fd1c8025564-web-config\") pod \"alertmanager-main-0\" (UID: \"d060b445-2d60-481c-a52d-0fd1c8025564\") " pod="openshift-monitoring/alertmanager-main-0" Jul 09 15:33:05.482459 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:33:05.482384 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/d060b445-2d60-481c-a52d-0fd1c8025564-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"d060b445-2d60-481c-a52d-0fd1c8025564\") " pod="openshift-monitoring/alertmanager-main-0" Jul 09 15:33:05.482459 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:33:05.482433 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/d060b445-2d60-481c-a52d-0fd1c8025564-tls-assets\") pod \"alertmanager-main-0\" (UID: \"d060b445-2d60-481c-a52d-0fd1c8025564\") " pod="openshift-monitoring/alertmanager-main-0" Jul 09 15:33:05.482459 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:33:05.482461 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/d060b445-2d60-481c-a52d-0fd1c8025564-config-out\") pod \"alertmanager-main-0\" (UID: \"d060b445-2d60-481c-a52d-0fd1c8025564\") " pod="openshift-monitoring/alertmanager-main-0" Jul 09 15:33:05.482865 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:33:05.482495 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/d060b445-2d60-481c-a52d-0fd1c8025564-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"d060b445-2d60-481c-a52d-0fd1c8025564\") " pod="openshift-monitoring/alertmanager-main-0" Jul 09 15:33:05.482865 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:33:05.482551 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/d060b445-2d60-481c-a52d-0fd1c8025564-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"d060b445-2d60-481c-a52d-0fd1c8025564\") " pod="openshift-monitoring/alertmanager-main-0" Jul 09 15:33:05.482865 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:33:05.482587 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/d060b445-2d60-481c-a52d-0fd1c8025564-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"d060b445-2d60-481c-a52d-0fd1c8025564\") " pod="openshift-monitoring/alertmanager-main-0" Jul 09 15:33:05.483518 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:33:05.483156 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/d060b445-2d60-481c-a52d-0fd1c8025564-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"d060b445-2d60-481c-a52d-0fd1c8025564\") " pod="openshift-monitoring/alertmanager-main-0" Jul 09 15:33:05.483518 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:33:05.483339 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d060b445-2d60-481c-a52d-0fd1c8025564-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"d060b445-2d60-481c-a52d-0fd1c8025564\") " pod="openshift-monitoring/alertmanager-main-0" Jul 09 15:33:05.483920 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:33:05.483899 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/d060b445-2d60-481c-a52d-0fd1c8025564-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"d060b445-2d60-481c-a52d-0fd1c8025564\") " pod="openshift-monitoring/alertmanager-main-0" Jul 09 15:33:05.485667 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:33:05.485637 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/d060b445-2d60-481c-a52d-0fd1c8025564-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"d060b445-2d60-481c-a52d-0fd1c8025564\") " pod="openshift-monitoring/alertmanager-main-0" Jul 09 15:33:05.486291 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:33:05.485999 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/d060b445-2d60-481c-a52d-0fd1c8025564-tls-assets\") pod \"alertmanager-main-0\" (UID: \"d060b445-2d60-481c-a52d-0fd1c8025564\") " pod="openshift-monitoring/alertmanager-main-0" Jul 09 15:33:05.486291 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:33:05.486005 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/d060b445-2d60-481c-a52d-0fd1c8025564-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"d060b445-2d60-481c-a52d-0fd1c8025564\") " pod="openshift-monitoring/alertmanager-main-0" Jul 09 15:33:05.486291 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:33:05.486156 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/d060b445-2d60-481c-a52d-0fd1c8025564-web-config\") pod \"alertmanager-main-0\" (UID: \"d060b445-2d60-481c-a52d-0fd1c8025564\") " pod="openshift-monitoring/alertmanager-main-0" Jul 09 15:33:05.486291 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:33:05.486243 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/d060b445-2d60-481c-a52d-0fd1c8025564-config-out\") pod \"alertmanager-main-0\" (UID: \"d060b445-2d60-481c-a52d-0fd1c8025564\") " pod="openshift-monitoring/alertmanager-main-0" Jul 09 15:33:05.486551 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:33:05.486491 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/d060b445-2d60-481c-a52d-0fd1c8025564-config-volume\") pod \"alertmanager-main-0\" (UID: \"d060b445-2d60-481c-a52d-0fd1c8025564\") " pod="openshift-monitoring/alertmanager-main-0" Jul 09 15:33:05.486551 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:33:05.486518 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/d060b445-2d60-481c-a52d-0fd1c8025564-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"d060b445-2d60-481c-a52d-0fd1c8025564\") " pod="openshift-monitoring/alertmanager-main-0" Jul 09 15:33:05.487068 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:33:05.487046 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/d060b445-2d60-481c-a52d-0fd1c8025564-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"d060b445-2d60-481c-a52d-0fd1c8025564\") " pod="openshift-monitoring/alertmanager-main-0" Jul 09 15:33:05.487282 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:33:05.487267 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/d060b445-2d60-481c-a52d-0fd1c8025564-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"d060b445-2d60-481c-a52d-0fd1c8025564\") " pod="openshift-monitoring/alertmanager-main-0" Jul 09 15:33:05.500891 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:33:05.500857 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tjhrd\" (UniqueName: \"kubernetes.io/projected/d060b445-2d60-481c-a52d-0fd1c8025564-kube-api-access-tjhrd\") pod \"alertmanager-main-0\" (UID: \"d060b445-2d60-481c-a52d-0fd1c8025564\") " pod="openshift-monitoring/alertmanager-main-0" Jul 09 15:33:05.538834 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:33:05.538797 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Jul 09 15:33:05.713197 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:33:05.713168 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Jul 09 15:33:05.714341 ip-10-0-133-183 kubenswrapper[2573]: W0709 15:33:05.714314 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd060b445_2d60_481c_a52d_0fd1c8025564.slice/crio-6e4c5f0baf58cf17fb99465b2233ba0620bbfce17cca4793ca146f18e17821a3 WatchSource:0}: Error finding container 6e4c5f0baf58cf17fb99465b2233ba0620bbfce17cca4793ca146f18e17821a3: Status 404 returned error can't find the container with id 6e4c5f0baf58cf17fb99465b2233ba0620bbfce17cca4793ca146f18e17821a3 Jul 09 15:33:06.300307 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:33:06.300267 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"d060b445-2d60-481c-a52d-0fd1c8025564","Type":"ContainerStarted","Data":"6e4c5f0baf58cf17fb99465b2233ba0620bbfce17cca4793ca146f18e17821a3"} Jul 09 15:33:07.305773 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:33:07.305682 2573 generic.go:356] "Generic (PLEG): container finished" podID="1886ffd8-a444-41ce-803c-58c58160faff" containerID="6a3418fd1483e689ab32baf467c1a26581a4b7d0c4d80fc222737187cee1f306" exitCode=0 Jul 09 15:33:07.306220 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:33:07.305780 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-4jbr9" event={"ID":"1886ffd8-a444-41ce-803c-58c58160faff","Type":"ContainerDied","Data":"6a3418fd1483e689ab32baf467c1a26581a4b7d0c4d80fc222737187cee1f306"} Jul 09 15:33:07.307325 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:33:07.307303 2573 generic.go:356] "Generic (PLEG): container finished" podID="d060b445-2d60-481c-a52d-0fd1c8025564" containerID="64549006016ade6298cbd43b84715f6dd691aaec4342e537cf364ea3a5b57a2f" exitCode=0 Jul 09 15:33:07.307425 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:33:07.307339 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"d060b445-2d60-481c-a52d-0fd1c8025564","Type":"ContainerDied","Data":"64549006016ade6298cbd43b84715f6dd691aaec4342e537cf364ea3a5b57a2f"} Jul 09 15:33:08.312953 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:33:08.312908 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-4jbr9" event={"ID":"1886ffd8-a444-41ce-803c-58c58160faff","Type":"ContainerStarted","Data":"94d4bb3d1132cdbc90d664200de93e9f95228eb5c9e3f27931a80b7e4c9a539e"} Jul 09 15:33:08.313335 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:33:08.312983 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-4jbr9" event={"ID":"1886ffd8-a444-41ce-803c-58c58160faff","Type":"ContainerStarted","Data":"8854e860b1c51a114cff6ea8e7793898fa92bd514a5607a945317a3e8ebb3318"} Jul 09 15:33:08.336820 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:33:08.336765 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-4jbr9" podStartSLOduration=3.271383072 podStartE2EDuration="5.336747664s" totalImagesPullingTime="2.000232404s" totalInitContainerRuntime="65.132188ms" isStatefulPod=true podCreationTimestamp="2026-07-09 15:33:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 15:33:08.335355434 +0000 UTC m=+110.056862305" watchObservedRunningTime="2026-07-09 15:33:08.336747664 +0000 UTC m=+110.058254511" Jul 09 15:33:08.507721 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:33:08.507676 2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/metrics-server-68cc7b6dfc-pqtq8"] Jul 09 15:33:08.510997 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:33:08.510960 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-68cc7b6dfc-pqtq8" Jul 09 15:33:08.514537 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:33:08.514510 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-tls\"" Jul 09 15:33:08.514706 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:33:08.514553 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-e86unl4isqhm1\"" Jul 09 15:33:08.514706 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:33:08.514561 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-client-certs\"" Jul 09 15:33:08.514706 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:33:08.514684 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-server-audit-profiles\"" Jul 09 15:33:08.514972 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:33:08.514935 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-dockercfg-w5gx9\"" Jul 09 15:33:08.515032 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:33:08.514973 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\"" Jul 09 15:33:08.520497 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:33:08.520473 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-68cc7b6dfc-pqtq8"] Jul 09 15:33:08.612036 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:33:08.611923 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/b98b23c1-1d45-4d6b-83cf-579c250605b7-secret-metrics-server-tls\") pod \"metrics-server-68cc7b6dfc-pqtq8\" (UID: \"b98b23c1-1d45-4d6b-83cf-579c250605b7\") " pod="openshift-monitoring/metrics-server-68cc7b6dfc-pqtq8" Jul 09 15:33:08.612036 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:33:08.611978 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-882fr\" (UniqueName: \"kubernetes.io/projected/b98b23c1-1d45-4d6b-83cf-579c250605b7-kube-api-access-882fr\") pod \"metrics-server-68cc7b6dfc-pqtq8\" (UID: \"b98b23c1-1d45-4d6b-83cf-579c250605b7\") " pod="openshift-monitoring/metrics-server-68cc7b6dfc-pqtq8" Jul 09 15:33:08.612258 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:33:08.612046 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b98b23c1-1d45-4d6b-83cf-579c250605b7-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-68cc7b6dfc-pqtq8\" (UID: \"b98b23c1-1d45-4d6b-83cf-579c250605b7\") " pod="openshift-monitoring/metrics-server-68cc7b6dfc-pqtq8" Jul 09 15:33:08.612258 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:33:08.612078 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/b98b23c1-1d45-4d6b-83cf-579c250605b7-secret-metrics-server-client-certs\") pod \"metrics-server-68cc7b6dfc-pqtq8\" (UID: \"b98b23c1-1d45-4d6b-83cf-579c250605b7\") " pod="openshift-monitoring/metrics-server-68cc7b6dfc-pqtq8" Jul 09 15:33:08.612258 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:33:08.612105 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/b98b23c1-1d45-4d6b-83cf-579c250605b7-audit-log\") pod \"metrics-server-68cc7b6dfc-pqtq8\" (UID: \"b98b23c1-1d45-4d6b-83cf-579c250605b7\") " pod="openshift-monitoring/metrics-server-68cc7b6dfc-pqtq8" Jul 09 15:33:08.612258 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:33:08.612139 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/b98b23c1-1d45-4d6b-83cf-579c250605b7-client-ca-bundle\") pod \"metrics-server-68cc7b6dfc-pqtq8\" (UID: \"b98b23c1-1d45-4d6b-83cf-579c250605b7\") " pod="openshift-monitoring/metrics-server-68cc7b6dfc-pqtq8" Jul 09 15:33:08.612258 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:33:08.612176 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/b98b23c1-1d45-4d6b-83cf-579c250605b7-metrics-server-audit-profiles\") pod \"metrics-server-68cc7b6dfc-pqtq8\" (UID: \"b98b23c1-1d45-4d6b-83cf-579c250605b7\") " pod="openshift-monitoring/metrics-server-68cc7b6dfc-pqtq8" Jul 09 15:33:08.668288 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:33:08.667646 2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/monitoring-plugin-5f68df7f58-47x4v"] Jul 09 15:33:08.671307 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:33:08.671279 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-5f68df7f58-47x4v" Jul 09 15:33:08.674002 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:33:08.673978 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-cert\"" Jul 09 15:33:08.674142 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:33:08.674108 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-dockercfg-f6wsp\"" Jul 09 15:33:08.683459 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:33:08.683405 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-5f68df7f58-47x4v"] Jul 09 15:33:08.712728 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:33:08.712688 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/b98b23c1-1d45-4d6b-83cf-579c250605b7-metrics-server-audit-profiles\") pod \"metrics-server-68cc7b6dfc-pqtq8\" (UID: \"b98b23c1-1d45-4d6b-83cf-579c250605b7\") " pod="openshift-monitoring/metrics-server-68cc7b6dfc-pqtq8" Jul 09 15:33:08.712948 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:33:08.712765 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/b98b23c1-1d45-4d6b-83cf-579c250605b7-secret-metrics-server-tls\") pod \"metrics-server-68cc7b6dfc-pqtq8\" (UID: \"b98b23c1-1d45-4d6b-83cf-579c250605b7\") " pod="openshift-monitoring/metrics-server-68cc7b6dfc-pqtq8" Jul 09 15:33:08.712948 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:33:08.712788 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-882fr\" (UniqueName: \"kubernetes.io/projected/b98b23c1-1d45-4d6b-83cf-579c250605b7-kube-api-access-882fr\") pod \"metrics-server-68cc7b6dfc-pqtq8\" (UID: \"b98b23c1-1d45-4d6b-83cf-579c250605b7\") " pod="openshift-monitoring/metrics-server-68cc7b6dfc-pqtq8" Jul 09 15:33:08.712948 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:33:08.712836 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b98b23c1-1d45-4d6b-83cf-579c250605b7-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-68cc7b6dfc-pqtq8\" (UID: \"b98b23c1-1d45-4d6b-83cf-579c250605b7\") " pod="openshift-monitoring/metrics-server-68cc7b6dfc-pqtq8" Jul 09 15:33:08.712948 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:33:08.712864 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/b98b23c1-1d45-4d6b-83cf-579c250605b7-secret-metrics-server-client-certs\") pod \"metrics-server-68cc7b6dfc-pqtq8\" (UID: \"b98b23c1-1d45-4d6b-83cf-579c250605b7\") " pod="openshift-monitoring/metrics-server-68cc7b6dfc-pqtq8" Jul 09 15:33:08.712948 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:33:08.712915 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/b98b23c1-1d45-4d6b-83cf-579c250605b7-audit-log\") pod \"metrics-server-68cc7b6dfc-pqtq8\" (UID: \"b98b23c1-1d45-4d6b-83cf-579c250605b7\") " pod="openshift-monitoring/metrics-server-68cc7b6dfc-pqtq8" Jul 09 15:33:08.712948 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:33:08.712947 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/b98b23c1-1d45-4d6b-83cf-579c250605b7-client-ca-bundle\") pod \"metrics-server-68cc7b6dfc-pqtq8\" (UID: \"b98b23c1-1d45-4d6b-83cf-579c250605b7\") " pod="openshift-monitoring/metrics-server-68cc7b6dfc-pqtq8" Jul 09 15:33:08.713407 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:33:08.713381 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/b98b23c1-1d45-4d6b-83cf-579c250605b7-audit-log\") pod \"metrics-server-68cc7b6dfc-pqtq8\" (UID: \"b98b23c1-1d45-4d6b-83cf-579c250605b7\") " pod="openshift-monitoring/metrics-server-68cc7b6dfc-pqtq8" Jul 09 15:33:08.713694 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:33:08.713640 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b98b23c1-1d45-4d6b-83cf-579c250605b7-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-68cc7b6dfc-pqtq8\" (UID: \"b98b23c1-1d45-4d6b-83cf-579c250605b7\") " pod="openshift-monitoring/metrics-server-68cc7b6dfc-pqtq8" Jul 09 15:33:08.713987 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:33:08.713929 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/b98b23c1-1d45-4d6b-83cf-579c250605b7-metrics-server-audit-profiles\") pod \"metrics-server-68cc7b6dfc-pqtq8\" (UID: \"b98b23c1-1d45-4d6b-83cf-579c250605b7\") " pod="openshift-monitoring/metrics-server-68cc7b6dfc-pqtq8" Jul 09 15:33:08.715989 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:33:08.715946 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/b98b23c1-1d45-4d6b-83cf-579c250605b7-client-ca-bundle\") pod \"metrics-server-68cc7b6dfc-pqtq8\" (UID: \"b98b23c1-1d45-4d6b-83cf-579c250605b7\") " pod="openshift-monitoring/metrics-server-68cc7b6dfc-pqtq8" Jul 09 15:33:08.716094 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:33:08.716060 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/b98b23c1-1d45-4d6b-83cf-579c250605b7-secret-metrics-server-client-certs\") pod \"metrics-server-68cc7b6dfc-pqtq8\" (UID: \"b98b23c1-1d45-4d6b-83cf-579c250605b7\") " pod="openshift-monitoring/metrics-server-68cc7b6dfc-pqtq8" Jul 09 15:33:08.716186 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:33:08.716165 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/b98b23c1-1d45-4d6b-83cf-579c250605b7-secret-metrics-server-tls\") pod \"metrics-server-68cc7b6dfc-pqtq8\" (UID: \"b98b23c1-1d45-4d6b-83cf-579c250605b7\") " pod="openshift-monitoring/metrics-server-68cc7b6dfc-pqtq8" Jul 09 15:33:08.728890 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:33:08.728845 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-882fr\" (UniqueName: \"kubernetes.io/projected/b98b23c1-1d45-4d6b-83cf-579c250605b7-kube-api-access-882fr\") pod \"metrics-server-68cc7b6dfc-pqtq8\" (UID: \"b98b23c1-1d45-4d6b-83cf-579c250605b7\") " pod="openshift-monitoring/metrics-server-68cc7b6dfc-pqtq8" Jul 09 15:33:08.813961 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:33:08.813926 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q9nc4\" (UniqueName: \"kubernetes.io/projected/07b98dec-386e-4afc-9345-8e8c2c6f9642-kube-api-access-q9nc4\") pod \"monitoring-plugin-5f68df7f58-47x4v\" (UID: \"07b98dec-386e-4afc-9345-8e8c2c6f9642\") " pod="openshift-monitoring/monitoring-plugin-5f68df7f58-47x4v" Jul 09 15:33:08.814124 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:33:08.813980 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/07b98dec-386e-4afc-9345-8e8c2c6f9642-monitoring-plugin-cert\") pod \"monitoring-plugin-5f68df7f58-47x4v\" (UID: \"07b98dec-386e-4afc-9345-8e8c2c6f9642\") " pod="openshift-monitoring/monitoring-plugin-5f68df7f58-47x4v" Jul 09 15:33:08.822784 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:33:08.822749 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-68cc7b6dfc-pqtq8" Jul 09 15:33:08.914400 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:33:08.914320 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q9nc4\" (UniqueName: \"kubernetes.io/projected/07b98dec-386e-4afc-9345-8e8c2c6f9642-kube-api-access-q9nc4\") pod \"monitoring-plugin-5f68df7f58-47x4v\" (UID: \"07b98dec-386e-4afc-9345-8e8c2c6f9642\") " pod="openshift-monitoring/monitoring-plugin-5f68df7f58-47x4v" Jul 09 15:33:08.914400 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:33:08.914363 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/07b98dec-386e-4afc-9345-8e8c2c6f9642-monitoring-plugin-cert\") pod \"monitoring-plugin-5f68df7f58-47x4v\" (UID: \"07b98dec-386e-4afc-9345-8e8c2c6f9642\") " pod="openshift-monitoring/monitoring-plugin-5f68df7f58-47x4v" Jul 09 15:33:08.914621 ip-10-0-133-183 kubenswrapper[2573]: E0709 15:33:08.914486 2573 secret.go:189] Couldn't get secret openshift-monitoring/monitoring-plugin-cert: secret "monitoring-plugin-cert" not found Jul 09 15:33:08.914621 ip-10-0-133-183 kubenswrapper[2573]: E0709 15:33:08.914555 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/07b98dec-386e-4afc-9345-8e8c2c6f9642-monitoring-plugin-cert podName:07b98dec-386e-4afc-9345-8e8c2c6f9642 nodeName:}" failed. No retries permitted until 2026-07-09 15:33:09.414535039 +0000 UTC m=+111.136041886 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "monitoring-plugin-cert" (UniqueName: "kubernetes.io/secret/07b98dec-386e-4afc-9345-8e8c2c6f9642-monitoring-plugin-cert") pod "monitoring-plugin-5f68df7f58-47x4v" (UID: "07b98dec-386e-4afc-9345-8e8c2c6f9642") : secret "monitoring-plugin-cert" not found Jul 09 15:33:08.931767 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:33:08.931710 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-q9nc4\" (UniqueName: \"kubernetes.io/projected/07b98dec-386e-4afc-9345-8e8c2c6f9642-kube-api-access-q9nc4\") pod \"monitoring-plugin-5f68df7f58-47x4v\" (UID: \"07b98dec-386e-4afc-9345-8e8c2c6f9642\") " pod="openshift-monitoring/monitoring-plugin-5f68df7f58-47x4v" Jul 09 15:33:09.321867 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:33:09.321821 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-84bd68b896-q9km7" event={"ID":"9889ca25-29ca-4226-8c5a-4e2eb8040868","Type":"ContainerStarted","Data":"6915f60bea52d6d6435e4f3368dd68446235a7f84cdb90c5e6bb72072fef3bd7"} Jul 09 15:33:09.324213 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:33:09.324161 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"d060b445-2d60-481c-a52d-0fd1c8025564","Type":"ContainerStarted","Data":"968ac3b96a9afddfd93811cfcb2ae97ad782940af168ee6f47db4f0ed4f21799"} Jul 09 15:33:09.343870 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:33:09.343799 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-68cc7b6dfc-pqtq8"] Jul 09 15:33:09.347475 ip-10-0-133-183 kubenswrapper[2573]: W0709 15:33:09.347442 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb98b23c1_1d45_4d6b_83cf_579c250605b7.slice/crio-bf128788c7595d50f74a88fc1e44f3356e82cc8e8ea3507ee0303622b31efcb8 WatchSource:0}: Error finding container bf128788c7595d50f74a88fc1e44f3356e82cc8e8ea3507ee0303622b31efcb8: Status 404 returned error can't find the container with id bf128788c7595d50f74a88fc1e44f3356e82cc8e8ea3507ee0303622b31efcb8 Jul 09 15:33:09.351222 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:33:09.351175 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/openshift-state-metrics-84bd68b896-q9km7" podStartSLOduration=1.879463743 podStartE2EDuration="6.351157859s" totalImagesPullingTime="4.471694116s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 15:33:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 15:33:09.350233868 +0000 UTC m=+111.071740750" watchObservedRunningTime="2026-07-09 15:33:09.351157859 +0000 UTC m=+111.072664727" Jul 09 15:33:09.418298 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:33:09.418275 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/07b98dec-386e-4afc-9345-8e8c2c6f9642-monitoring-plugin-cert\") pod \"monitoring-plugin-5f68df7f58-47x4v\" (UID: \"07b98dec-386e-4afc-9345-8e8c2c6f9642\") " pod="openshift-monitoring/monitoring-plugin-5f68df7f58-47x4v" Jul 09 15:33:09.421058 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:33:09.421032 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/07b98dec-386e-4afc-9345-8e8c2c6f9642-monitoring-plugin-cert\") pod \"monitoring-plugin-5f68df7f58-47x4v\" (UID: \"07b98dec-386e-4afc-9345-8e8c2c6f9642\") " pod="openshift-monitoring/monitoring-plugin-5f68df7f58-47x4v" Jul 09 15:33:09.583635 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:33:09.583599 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-5f68df7f58-47x4v" Jul 09 15:33:09.719051 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:33:09.719010 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-5f68df7f58-47x4v"] Jul 09 15:33:09.721671 ip-10-0-133-183 kubenswrapper[2573]: W0709 15:33:09.721642 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod07b98dec_386e_4afc_9345_8e8c2c6f9642.slice/crio-f0aee9a978190c264fe7f7b7f24700c3a2f1303c1bfea10f01c5ce02739af169 WatchSource:0}: Error finding container f0aee9a978190c264fe7f7b7f24700c3a2f1303c1bfea10f01c5ce02739af169: Status 404 returned error can't find the container with id f0aee9a978190c264fe7f7b7f24700c3a2f1303c1bfea10f01c5ce02739af169 Jul 09 15:33:10.332591 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:33:10.332397 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-68cc7b6dfc-pqtq8" event={"ID":"b98b23c1-1d45-4d6b-83cf-579c250605b7","Type":"ContainerStarted","Data":"bf128788c7595d50f74a88fc1e44f3356e82cc8e8ea3507ee0303622b31efcb8"} Jul 09 15:33:10.335043 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:33:10.334961 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-5f68df7f58-47x4v" event={"ID":"07b98dec-386e-4afc-9345-8e8c2c6f9642","Type":"ContainerStarted","Data":"f0aee9a978190c264fe7f7b7f24700c3a2f1303c1bfea10f01c5ce02739af169"} Jul 09 15:33:10.338554 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:33:10.338385 2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Jul 09 15:33:10.342985 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:33:10.342745 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"d060b445-2d60-481c-a52d-0fd1c8025564","Type":"ContainerStarted","Data":"ede44ec66ad7f7461b9593a598a04368e991a98bedf99a69b3318f5d7ba6178e"} Jul 09 15:33:10.342985 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:33:10.342837 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"d060b445-2d60-481c-a52d-0fd1c8025564","Type":"ContainerStarted","Data":"614905de0091d4abdbb7b6c849224fed35f94f4329759a6b7f831148a7872542"} Jul 09 15:33:10.342985 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:33:10.342907 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"d060b445-2d60-481c-a52d-0fd1c8025564","Type":"ContainerStarted","Data":"4b7c9f674a42ac59eb85cf46a439dd2f74f2e9108169ed78237956934317d4ee"} Jul 09 15:33:10.342985 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:33:10.342951 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"d060b445-2d60-481c-a52d-0fd1c8025564","Type":"ContainerStarted","Data":"cc9b33d655f75df5aefc1d00594f69a8846d603f01bffcbed836335a0ee8dcdf"} Jul 09 15:33:10.343296 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:33:10.343103 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Jul 09 15:33:10.347703 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:33:10.347177 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\"" Jul 09 15:33:10.347703 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:33:10.347612 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\"" Jul 09 15:33:10.347703 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:33:10.347697 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\"" Jul 09 15:33:10.348102 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:33:10.347958 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\"" Jul 09 15:33:10.348335 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:33:10.348237 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-tg8ns\"" Jul 09 15:33:10.348793 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:33:10.348769 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-2v2hm3mjjrm9q\"" Jul 09 15:33:10.349084 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:33:10.349067 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\"" Jul 09 15:33:10.349265 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:33:10.349249 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\"" Jul 09 15:33:10.349498 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:33:10.349474 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\"" Jul 09 15:33:10.349686 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:33:10.349662 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\"" Jul 09 15:33:10.349904 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:33:10.349864 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\"" Jul 09 15:33:10.350291 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:33:10.350053 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\"" Jul 09 15:33:10.354077 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:33:10.353765 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\"" Jul 09 15:33:10.354077 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:33:10.353953 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\"" Jul 09 15:33:10.354499 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:33:10.354349 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\"" Jul 09 15:33:10.355801 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:33:10.355697 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\"" Jul 09 15:33:10.367898 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:33:10.367851 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Jul 09 15:33:10.528468 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:33:10.528436 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/c219e7c1-044b-4e08-8455-362bec98f934-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"c219e7c1-044b-4e08-8455-362bec98f934\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 09 15:33:10.528657 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:33:10.528497 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/c219e7c1-044b-4e08-8455-362bec98f934-config\") pod \"prometheus-k8s-0\" (UID: \"c219e7c1-044b-4e08-8455-362bec98f934\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 09 15:33:10.528657 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:33:10.528531 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/c219e7c1-044b-4e08-8455-362bec98f934-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"c219e7c1-044b-4e08-8455-362bec98f934\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 09 15:33:10.528657 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:33:10.528553 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/c219e7c1-044b-4e08-8455-362bec98f934-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"c219e7c1-044b-4e08-8455-362bec98f934\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 09 15:33:10.528657 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:33:10.528582 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c219e7c1-044b-4e08-8455-362bec98f934-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"c219e7c1-044b-4e08-8455-362bec98f934\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 09 15:33:10.528657 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:33:10.528615 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/c219e7c1-044b-4e08-8455-362bec98f934-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"c219e7c1-044b-4e08-8455-362bec98f934\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 09 15:33:10.528657 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:33:10.528636 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c219e7c1-044b-4e08-8455-362bec98f934-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"c219e7c1-044b-4e08-8455-362bec98f934\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 09 15:33:10.528993 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:33:10.528656 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/c219e7c1-044b-4e08-8455-362bec98f934-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"c219e7c1-044b-4e08-8455-362bec98f934\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 09 15:33:10.528993 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:33:10.528688 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/c219e7c1-044b-4e08-8455-362bec98f934-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"c219e7c1-044b-4e08-8455-362bec98f934\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 09 15:33:10.528993 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:33:10.528725 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/c219e7c1-044b-4e08-8455-362bec98f934-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"c219e7c1-044b-4e08-8455-362bec98f934\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 09 15:33:10.528993 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:33:10.528781 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/c219e7c1-044b-4e08-8455-362bec98f934-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"c219e7c1-044b-4e08-8455-362bec98f934\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 09 15:33:10.528993 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:33:10.528845 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/c219e7c1-044b-4e08-8455-362bec98f934-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"c219e7c1-044b-4e08-8455-362bec98f934\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 09 15:33:10.528993 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:33:10.528899 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/c219e7c1-044b-4e08-8455-362bec98f934-web-config\") pod \"prometheus-k8s-0\" (UID: \"c219e7c1-044b-4e08-8455-362bec98f934\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 09 15:33:10.528993 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:33:10.528950 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/c219e7c1-044b-4e08-8455-362bec98f934-config-out\") pod \"prometheus-k8s-0\" (UID: \"c219e7c1-044b-4e08-8455-362bec98f934\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 09 15:33:10.528993 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:33:10.528974 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8b4jc\" (UniqueName: \"kubernetes.io/projected/c219e7c1-044b-4e08-8455-362bec98f934-kube-api-access-8b4jc\") pod \"prometheus-k8s-0\" (UID: \"c219e7c1-044b-4e08-8455-362bec98f934\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 09 15:33:10.529349 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:33:10.529006 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c219e7c1-044b-4e08-8455-362bec98f934-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"c219e7c1-044b-4e08-8455-362bec98f934\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 09 15:33:10.529349 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:33:10.529035 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/c219e7c1-044b-4e08-8455-362bec98f934-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"c219e7c1-044b-4e08-8455-362bec98f934\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 09 15:33:10.529349 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:33:10.529077 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/c219e7c1-044b-4e08-8455-362bec98f934-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"c219e7c1-044b-4e08-8455-362bec98f934\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 09 15:33:10.529349 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:33:10.529100 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/c219e7c1-044b-4e08-8455-362bec98f934-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"c219e7c1-044b-4e08-8455-362bec98f934\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 09 15:33:10.529349 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:33:10.529143 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/c219e7c1-044b-4e08-8455-362bec98f934-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"c219e7c1-044b-4e08-8455-362bec98f934\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 09 15:33:10.630107 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:33:10.630011 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/c219e7c1-044b-4e08-8455-362bec98f934-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"c219e7c1-044b-4e08-8455-362bec98f934\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 09 15:33:10.630107 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:33:10.630070 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c219e7c1-044b-4e08-8455-362bec98f934-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"c219e7c1-044b-4e08-8455-362bec98f934\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 09 15:33:10.630107 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:33:10.630101 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/c219e7c1-044b-4e08-8455-362bec98f934-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"c219e7c1-044b-4e08-8455-362bec98f934\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 09 15:33:10.630374 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:33:10.630134 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/c219e7c1-044b-4e08-8455-362bec98f934-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"c219e7c1-044b-4e08-8455-362bec98f934\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 09 15:33:10.630374 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:33:10.630167 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/c219e7c1-044b-4e08-8455-362bec98f934-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"c219e7c1-044b-4e08-8455-362bec98f934\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 09 15:33:10.630374 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:33:10.630204 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/c219e7c1-044b-4e08-8455-362bec98f934-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"c219e7c1-044b-4e08-8455-362bec98f934\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 09 15:33:10.630374 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:33:10.630230 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/c219e7c1-044b-4e08-8455-362bec98f934-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"c219e7c1-044b-4e08-8455-362bec98f934\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 09 15:33:10.630374 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:33:10.630262 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/c219e7c1-044b-4e08-8455-362bec98f934-web-config\") pod \"prometheus-k8s-0\" (UID: \"c219e7c1-044b-4e08-8455-362bec98f934\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 09 15:33:10.630374 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:33:10.630295 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/c219e7c1-044b-4e08-8455-362bec98f934-config-out\") pod \"prometheus-k8s-0\" (UID: \"c219e7c1-044b-4e08-8455-362bec98f934\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 09 15:33:10.630374 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:33:10.630316 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8b4jc\" (UniqueName: \"kubernetes.io/projected/c219e7c1-044b-4e08-8455-362bec98f934-kube-api-access-8b4jc\") pod \"prometheus-k8s-0\" (UID: \"c219e7c1-044b-4e08-8455-362bec98f934\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 09 15:33:10.630374 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:33:10.630342 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c219e7c1-044b-4e08-8455-362bec98f934-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"c219e7c1-044b-4e08-8455-362bec98f934\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 09 15:33:10.630374 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:33:10.630368 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/c219e7c1-044b-4e08-8455-362bec98f934-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"c219e7c1-044b-4e08-8455-362bec98f934\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 09 15:33:10.630834 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:33:10.630406 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/c219e7c1-044b-4e08-8455-362bec98f934-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"c219e7c1-044b-4e08-8455-362bec98f934\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 09 15:33:10.630834 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:33:10.630428 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/c219e7c1-044b-4e08-8455-362bec98f934-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"c219e7c1-044b-4e08-8455-362bec98f934\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 09 15:33:10.630834 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:33:10.630460 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/c219e7c1-044b-4e08-8455-362bec98f934-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"c219e7c1-044b-4e08-8455-362bec98f934\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 09 15:33:10.630834 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:33:10.630511 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/c219e7c1-044b-4e08-8455-362bec98f934-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"c219e7c1-044b-4e08-8455-362bec98f934\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 09 15:33:10.630834 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:33:10.630540 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/c219e7c1-044b-4e08-8455-362bec98f934-config\") pod \"prometheus-k8s-0\" (UID: \"c219e7c1-044b-4e08-8455-362bec98f934\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 09 15:33:10.630834 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:33:10.630578 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/c219e7c1-044b-4e08-8455-362bec98f934-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"c219e7c1-044b-4e08-8455-362bec98f934\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 09 15:33:10.630834 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:33:10.630601 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/c219e7c1-044b-4e08-8455-362bec98f934-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"c219e7c1-044b-4e08-8455-362bec98f934\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 09 15:33:10.630834 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:33:10.630634 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c219e7c1-044b-4e08-8455-362bec98f934-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"c219e7c1-044b-4e08-8455-362bec98f934\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 09 15:33:10.630834 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:33:10.630757 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/c219e7c1-044b-4e08-8455-362bec98f934-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"c219e7c1-044b-4e08-8455-362bec98f934\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 09 15:33:10.634228 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:33:10.631035 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/c219e7c1-044b-4e08-8455-362bec98f934-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"c219e7c1-044b-4e08-8455-362bec98f934\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 09 15:33:10.634228 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:33:10.631092 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c219e7c1-044b-4e08-8455-362bec98f934-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"c219e7c1-044b-4e08-8455-362bec98f934\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 09 15:33:10.634228 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:33:10.631568 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c219e7c1-044b-4e08-8455-362bec98f934-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"c219e7c1-044b-4e08-8455-362bec98f934\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 09 15:33:10.635731 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:33:10.635313 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/c219e7c1-044b-4e08-8455-362bec98f934-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"c219e7c1-044b-4e08-8455-362bec98f934\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 09 15:33:10.636354 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:33:10.636035 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/c219e7c1-044b-4e08-8455-362bec98f934-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"c219e7c1-044b-4e08-8455-362bec98f934\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 09 15:33:10.639520 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:33:10.638035 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/c219e7c1-044b-4e08-8455-362bec98f934-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"c219e7c1-044b-4e08-8455-362bec98f934\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 09 15:33:10.639520 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:33:10.638731 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/c219e7c1-044b-4e08-8455-362bec98f934-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"c219e7c1-044b-4e08-8455-362bec98f934\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 09 15:33:10.639520 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:33:10.639120 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c219e7c1-044b-4e08-8455-362bec98f934-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"c219e7c1-044b-4e08-8455-362bec98f934\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 09 15:33:10.640165 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:33:10.639754 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/c219e7c1-044b-4e08-8455-362bec98f934-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"c219e7c1-044b-4e08-8455-362bec98f934\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 09 15:33:10.640165 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:33:10.639805 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/c219e7c1-044b-4e08-8455-362bec98f934-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"c219e7c1-044b-4e08-8455-362bec98f934\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 09 15:33:10.640165 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:33:10.640046 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/c219e7c1-044b-4e08-8455-362bec98f934-config-out\") pod \"prometheus-k8s-0\" (UID: \"c219e7c1-044b-4e08-8455-362bec98f934\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 09 15:33:10.640165 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:33:10.640138 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/c219e7c1-044b-4e08-8455-362bec98f934-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"c219e7c1-044b-4e08-8455-362bec98f934\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 09 15:33:10.640428 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:33:10.640340 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/c219e7c1-044b-4e08-8455-362bec98f934-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"c219e7c1-044b-4e08-8455-362bec98f934\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 09 15:33:10.640637 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:33:10.640614 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/c219e7c1-044b-4e08-8455-362bec98f934-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"c219e7c1-044b-4e08-8455-362bec98f934\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 09 15:33:10.640974 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:33:10.640945 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/c219e7c1-044b-4e08-8455-362bec98f934-config\") pod \"prometheus-k8s-0\" (UID: \"c219e7c1-044b-4e08-8455-362bec98f934\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 09 15:33:10.641315 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:33:10.641290 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/c219e7c1-044b-4e08-8455-362bec98f934-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"c219e7c1-044b-4e08-8455-362bec98f934\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 09 15:33:10.642374 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:33:10.642334 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/c219e7c1-044b-4e08-8455-362bec98f934-web-config\") pod \"prometheus-k8s-0\" (UID: \"c219e7c1-044b-4e08-8455-362bec98f934\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 09 15:33:10.643496 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:33:10.643471 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/c219e7c1-044b-4e08-8455-362bec98f934-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"c219e7c1-044b-4e08-8455-362bec98f934\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 09 15:33:10.653102 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:33:10.653075 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8b4jc\" (UniqueName: \"kubernetes.io/projected/c219e7c1-044b-4e08-8455-362bec98f934-kube-api-access-8b4jc\") pod \"prometheus-k8s-0\" (UID: \"c219e7c1-044b-4e08-8455-362bec98f934\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 09 15:33:10.658992 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:33:10.658952 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Jul 09 15:33:11.610770 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:33:11.610656 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Jul 09 15:33:11.613038 ip-10-0-133-183 kubenswrapper[2573]: W0709 15:33:11.612999 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc219e7c1_044b_4e08_8455_362bec98f934.slice/crio-6ff79a171f32bef2ccf5f9b365a30bffc4e198f048c49a989f2d9d5f5e99fe76 WatchSource:0}: Error finding container 6ff79a171f32bef2ccf5f9b365a30bffc4e198f048c49a989f2d9d5f5e99fe76: Status 404 returned error can't find the container with id 6ff79a171f32bef2ccf5f9b365a30bffc4e198f048c49a989f2d9d5f5e99fe76 Jul 09 15:33:12.351653 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:33:12.351612 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-68cc7b6dfc-pqtq8" event={"ID":"b98b23c1-1d45-4d6b-83cf-579c250605b7","Type":"ContainerStarted","Data":"86a11d555c3453f2e6acee80a85d31cebdf86114de581f848651f972987a82b5"} Jul 09 15:33:12.355024 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:33:12.354994 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"d060b445-2d60-481c-a52d-0fd1c8025564","Type":"ContainerStarted","Data":"0904490f9c40efb6a479415443b226ac5c09c4465a2350753585e82093498593"} Jul 09 15:33:12.356515 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:33:12.356492 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-5f68df7f58-47x4v" event={"ID":"07b98dec-386e-4afc-9345-8e8c2c6f9642","Type":"ContainerStarted","Data":"429fbd5aaea014882554032c5de9d48c1dbe5a3569899b6b0365749cb94721e3"} Jul 09 15:33:12.356630 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:33:12.356611 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/monitoring-plugin-5f68df7f58-47x4v" Jul 09 15:33:12.358192 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:33:12.358171 2573 generic.go:356] "Generic (PLEG): container finished" podID="c219e7c1-044b-4e08-8455-362bec98f934" containerID="566d490a703f9d0017f1d0e52e080147bb6dcdf54908050b3d42c3f7cffadf78" exitCode=0 Jul 09 15:33:12.358303 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:33:12.358194 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"c219e7c1-044b-4e08-8455-362bec98f934","Type":"ContainerDied","Data":"566d490a703f9d0017f1d0e52e080147bb6dcdf54908050b3d42c3f7cffadf78"} Jul 09 15:33:12.358364 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:33:12.358295 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"c219e7c1-044b-4e08-8455-362bec98f934","Type":"ContainerStarted","Data":"6ff79a171f32bef2ccf5f9b365a30bffc4e198f048c49a989f2d9d5f5e99fe76"} Jul 09 15:33:12.362352 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:33:12.362334 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/monitoring-plugin-5f68df7f58-47x4v" Jul 09 15:33:12.375134 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:33:12.375090 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/metrics-server-68cc7b6dfc-pqtq8" podStartSLOduration=2.277199102 podStartE2EDuration="4.375074112s" totalImagesPullingTime="2.09787501s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 15:33:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 15:33:12.374946591 +0000 UTC m=+114.096453459" watchObservedRunningTime="2026-07-09 15:33:12.375074112 +0000 UTC m=+114.096580981" Jul 09 15:33:12.405859 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:33:12.405808 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/alertmanager-main-0" podStartSLOduration=2.199438054 podStartE2EDuration="7.405791376s" totalImagesPullingTime="5.187259686s" totalInitContainerRuntime="19.093636ms" isStatefulPod=false podCreationTimestamp="2026-07-09 15:33:05 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 15:33:12.4041859 +0000 UTC m=+114.125692793" watchObservedRunningTime="2026-07-09 15:33:12.405791376 +0000 UTC m=+114.127298245" Jul 09 15:33:12.428096 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:33:12.428043 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/monitoring-plugin-5f68df7f58-47x4v" podStartSLOduration=2.699591669 podStartE2EDuration="4.428024654s" totalImagesPullingTime="1.728432985s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 15:33:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 15:33:12.427331878 +0000 UTC m=+114.148838770" watchObservedRunningTime="2026-07-09 15:33:12.428024654 +0000 UTC m=+114.149531522" Jul 09 15:33:17.381283 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:33:17.381198 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"c219e7c1-044b-4e08-8455-362bec98f934","Type":"ContainerStarted","Data":"0190d660cf533ab6e7b7db7d34a4095cd189a6818eb82a05010001e83f3c715c"} Jul 09 15:33:17.381629 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:33:17.381298 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"c219e7c1-044b-4e08-8455-362bec98f934","Type":"ContainerStarted","Data":"620e990a63c03c39603ae71dff10ba5a27ea321f3c8034c705d217e5de21ddaf"} Jul 09 15:33:19.393596 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:33:19.393558 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"c219e7c1-044b-4e08-8455-362bec98f934","Type":"ContainerStarted","Data":"04f9177fc3da99c80718756d26156a5072d4629788fd4f4a3b63437d18160bca"} Jul 09 15:33:19.394022 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:33:19.393682 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"c219e7c1-044b-4e08-8455-362bec98f934","Type":"ContainerStarted","Data":"b33b0483cd110b9e9cef64575e959c75505a4e174e1351ee22a2c7e6c4085de9"} Jul 09 15:33:19.394022 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:33:19.393726 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"c219e7c1-044b-4e08-8455-362bec98f934","Type":"ContainerStarted","Data":"6c6f5166d7d178eed3d7fe162e026e8ab92d6f303e8fd5ec5cc493e8a10df941"} Jul 09 15:33:19.802451 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:33:19.802418 2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-d6c6bcfbb-fjcg2"] Jul 09 15:33:20.401376 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:33:20.401337 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"c219e7c1-044b-4e08-8455-362bec98f934","Type":"ContainerStarted","Data":"93306c8519bd174aac1aa859b3353d2cbfd21dfa77232ad485ba0d941181dede"} Jul 09 15:33:20.445059 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:33:20.445000 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-k8s-0" podStartSLOduration=3.706094754 podStartE2EDuration="10.444982635s" totalImagesPullingTime="6.718226829s" totalInitContainerRuntime="20.661052ms" isStatefulPod=false podCreationTimestamp="2026-07-09 15:33:10 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 15:33:20.444303724 +0000 UTC m=+122.165810593" watchObservedRunningTime="2026-07-09 15:33:20.444982635 +0000 UTC m=+122.166489504" Jul 09 15:33:20.659327 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:33:20.659231 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-k8s-0" Jul 09 15:33:28.823591 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:33:28.823560 2573 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/metrics-server-68cc7b6dfc-pqtq8" Jul 09 15:33:28.824111 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:33:28.823608 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/metrics-server-68cc7b6dfc-pqtq8" Jul 09 15:33:44.823112 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:33:44.823074 2573 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-d6c6bcfbb-fjcg2" podUID="fc654672-12df-4a71-827f-1951234b78e8" containerName="console" containerID="cri-o://30fdc0ef2a5ca8a9c520688d93554199622d08dd2f4b5196a572abd21154e67c" gracePeriod=15 Jul 09 15:33:45.080365 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:33:45.080302 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-d6c6bcfbb-fjcg2" Jul 09 15:33:45.227226 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:33:45.227190 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/fc654672-12df-4a71-827f-1951234b78e8-service-ca\") pod \"fc654672-12df-4a71-827f-1951234b78e8\" (UID: \"fc654672-12df-4a71-827f-1951234b78e8\") " Jul 09 15:33:45.227439 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:33:45.227262 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/fc654672-12df-4a71-827f-1951234b78e8-trusted-ca-bundle\") pod \"fc654672-12df-4a71-827f-1951234b78e8\" (UID: \"fc654672-12df-4a71-827f-1951234b78e8\") " Jul 09 15:33:45.227439 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:33:45.227293 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7hh2b\" (UniqueName: \"kubernetes.io/projected/fc654672-12df-4a71-827f-1951234b78e8-kube-api-access-7hh2b\") pod \"fc654672-12df-4a71-827f-1951234b78e8\" (UID: \"fc654672-12df-4a71-827f-1951234b78e8\") " Jul 09 15:33:45.227439 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:33:45.227320 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/fc654672-12df-4a71-827f-1951234b78e8-console-config\") pod \"fc654672-12df-4a71-827f-1951234b78e8\" (UID: \"fc654672-12df-4a71-827f-1951234b78e8\") " Jul 09 15:33:45.227439 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:33:45.227349 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/fc654672-12df-4a71-827f-1951234b78e8-console-oauth-config\") pod \"fc654672-12df-4a71-827f-1951234b78e8\" (UID: \"fc654672-12df-4a71-827f-1951234b78e8\") " Jul 09 15:33:45.227439 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:33:45.227383 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/fc654672-12df-4a71-827f-1951234b78e8-oauth-serving-cert\") pod \"fc654672-12df-4a71-827f-1951234b78e8\" (UID: \"fc654672-12df-4a71-827f-1951234b78e8\") " Jul 09 15:33:45.227439 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:33:45.227414 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/fc654672-12df-4a71-827f-1951234b78e8-console-serving-cert\") pod \"fc654672-12df-4a71-827f-1951234b78e8\" (UID: \"fc654672-12df-4a71-827f-1951234b78e8\") " Jul 09 15:33:45.227766 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:33:45.227731 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fc654672-12df-4a71-827f-1951234b78e8-service-ca" (OuterVolumeSpecName: "service-ca") pod "fc654672-12df-4a71-827f-1951234b78e8" (UID: "fc654672-12df-4a71-827f-1951234b78e8"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 09 15:33:45.227856 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:33:45.227736 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fc654672-12df-4a71-827f-1951234b78e8-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "fc654672-12df-4a71-827f-1951234b78e8" (UID: "fc654672-12df-4a71-827f-1951234b78e8"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 09 15:33:45.227943 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:33:45.227864 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fc654672-12df-4a71-827f-1951234b78e8-console-config" (OuterVolumeSpecName: "console-config") pod "fc654672-12df-4a71-827f-1951234b78e8" (UID: "fc654672-12df-4a71-827f-1951234b78e8"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 09 15:33:45.227943 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:33:45.227870 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fc654672-12df-4a71-827f-1951234b78e8-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "fc654672-12df-4a71-827f-1951234b78e8" (UID: "fc654672-12df-4a71-827f-1951234b78e8"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 09 15:33:45.229807 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:33:45.229779 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fc654672-12df-4a71-827f-1951234b78e8-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "fc654672-12df-4a71-827f-1951234b78e8" (UID: "fc654672-12df-4a71-827f-1951234b78e8"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 09 15:33:45.230051 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:33:45.230032 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fc654672-12df-4a71-827f-1951234b78e8-kube-api-access-7hh2b" (OuterVolumeSpecName: "kube-api-access-7hh2b") pod "fc654672-12df-4a71-827f-1951234b78e8" (UID: "fc654672-12df-4a71-827f-1951234b78e8"). InnerVolumeSpecName "kube-api-access-7hh2b". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 09 15:33:45.230135 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:33:45.230115 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fc654672-12df-4a71-827f-1951234b78e8-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "fc654672-12df-4a71-827f-1951234b78e8" (UID: "fc654672-12df-4a71-827f-1951234b78e8"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 09 15:33:45.328436 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:33:45.328398 2573 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/fc654672-12df-4a71-827f-1951234b78e8-console-oauth-config\") on node \"ip-10-0-133-183.ec2.internal\" DevicePath \"\"" Jul 09 15:33:45.328436 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:33:45.328428 2573 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/fc654672-12df-4a71-827f-1951234b78e8-oauth-serving-cert\") on node \"ip-10-0-133-183.ec2.internal\" DevicePath \"\"" Jul 09 15:33:45.328436 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:33:45.328438 2573 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/fc654672-12df-4a71-827f-1951234b78e8-console-serving-cert\") on node \"ip-10-0-133-183.ec2.internal\" DevicePath \"\"" Jul 09 15:33:45.328436 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:33:45.328447 2573 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/fc654672-12df-4a71-827f-1951234b78e8-service-ca\") on node \"ip-10-0-133-183.ec2.internal\" DevicePath \"\"" Jul 09 15:33:45.328699 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:33:45.328454 2573 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/fc654672-12df-4a71-827f-1951234b78e8-trusted-ca-bundle\") on node \"ip-10-0-133-183.ec2.internal\" DevicePath \"\"" Jul 09 15:33:45.328699 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:33:45.328463 2573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7hh2b\" (UniqueName: \"kubernetes.io/projected/fc654672-12df-4a71-827f-1951234b78e8-kube-api-access-7hh2b\") on node \"ip-10-0-133-183.ec2.internal\" DevicePath \"\"" Jul 09 15:33:45.328699 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:33:45.328472 2573 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/fc654672-12df-4a71-827f-1951234b78e8-console-config\") on node \"ip-10-0-133-183.ec2.internal\" DevicePath \"\"" Jul 09 15:33:45.501402 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:33:45.501369 2573 generic.go:356] "Generic (PLEG): container finished" podID="fc654672-12df-4a71-827f-1951234b78e8" containerID="30fdc0ef2a5ca8a9c520688d93554199622d08dd2f4b5196a572abd21154e67c" exitCode=0 Jul 09 15:33:45.501571 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:33:45.501416 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-d6c6bcfbb-fjcg2" Jul 09 15:33:45.501571 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:33:45.501438 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-d6c6bcfbb-fjcg2" event={"ID":"fc654672-12df-4a71-827f-1951234b78e8","Type":"ContainerDied","Data":"30fdc0ef2a5ca8a9c520688d93554199622d08dd2f4b5196a572abd21154e67c"} Jul 09 15:33:45.501571 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:33:45.501504 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-d6c6bcfbb-fjcg2" event={"ID":"fc654672-12df-4a71-827f-1951234b78e8","Type":"ContainerDied","Data":"b3e6304b05152a6668ceea7e5548ccaaf1cc2bf2e23088167def150941bcba9e"} Jul 09 15:33:45.501571 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:33:45.501523 2573 scope.go:117] "RemoveContainer" containerID="30fdc0ef2a5ca8a9c520688d93554199622d08dd2f4b5196a572abd21154e67c" Jul 09 15:33:45.510283 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:33:45.510257 2573 scope.go:117] "RemoveContainer" containerID="30fdc0ef2a5ca8a9c520688d93554199622d08dd2f4b5196a572abd21154e67c" Jul 09 15:33:45.510541 ip-10-0-133-183 kubenswrapper[2573]: E0709 15:33:45.510519 2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"30fdc0ef2a5ca8a9c520688d93554199622d08dd2f4b5196a572abd21154e67c\": container with ID starting with 30fdc0ef2a5ca8a9c520688d93554199622d08dd2f4b5196a572abd21154e67c not found: ID does not exist" containerID="30fdc0ef2a5ca8a9c520688d93554199622d08dd2f4b5196a572abd21154e67c" Jul 09 15:33:45.510603 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:33:45.510553 2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"30fdc0ef2a5ca8a9c520688d93554199622d08dd2f4b5196a572abd21154e67c"} err="failed to get container status \"30fdc0ef2a5ca8a9c520688d93554199622d08dd2f4b5196a572abd21154e67c\": rpc error: code = NotFound desc = could not find container \"30fdc0ef2a5ca8a9c520688d93554199622d08dd2f4b5196a572abd21154e67c\": container with ID starting with 30fdc0ef2a5ca8a9c520688d93554199622d08dd2f4b5196a572abd21154e67c not found: ID does not exist" Jul 09 15:33:45.527127 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:33:45.527096 2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-d6c6bcfbb-fjcg2"] Jul 09 15:33:45.532562 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:33:45.532534 2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-d6c6bcfbb-fjcg2"] Jul 09 15:33:46.809783 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:33:46.809751 2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="fc654672-12df-4a71-827f-1951234b78e8" path="/var/lib/kubelet/pods/fc654672-12df-4a71-827f-1951234b78e8/volumes" Jul 09 15:33:48.828411 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:33:48.828381 2573 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/metrics-server-68cc7b6dfc-pqtq8" Jul 09 15:33:48.832142 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:33:48.832122 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/metrics-server-68cc7b6dfc-pqtq8" Jul 09 15:34:10.659836 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:34:10.659750 2573 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/prometheus-k8s-0" Jul 09 15:34:10.680152 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:34:10.680124 2573 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/prometheus-k8s-0" Jul 09 15:34:11.615962 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:34:11.615933 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-k8s-0" Jul 09 15:36:18.651130 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:36:18.651093 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-b7b9v_db22f88d-a953-488e-b384-67f991351f73/ovn-acl-logging/0.log" Jul 09 15:36:18.651967 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:36:18.651947 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-b7b9v_db22f88d-a953-488e-b384-67f991351f73/ovn-acl-logging/0.log" Jul 09 15:36:18.653571 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:36:18.653552 2573 kubelet.go:1648] "Image garbage collection succeeded" Jul 09 15:36:25.549070 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:36:25.549029 2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-lws-operator/openshift-lws-operator-fd8ccff4c-kqf2g"] Jul 09 15:36:25.551215 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:36:25.549341 2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fc654672-12df-4a71-827f-1951234b78e8" containerName="console" Jul 09 15:36:25.551215 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:36:25.549352 2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="fc654672-12df-4a71-827f-1951234b78e8" containerName="console" Jul 09 15:36:25.551215 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:36:25.549419 2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="fc654672-12df-4a71-827f-1951234b78e8" containerName="console" Jul 09 15:36:25.552027 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:36:25.552011 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-kqf2g" Jul 09 15:36:25.556907 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:36:25.556863 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"kube-root-ca.crt\"" Jul 09 15:36:25.557630 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:36:25.557608 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"openshift-lws-operator-dockercfg-g4f9d\"" Jul 09 15:36:25.557746 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:36:25.557610 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"openshift-service-ca.crt\"" Jul 09 15:36:25.561953 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:36:25.561932 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/openshift-lws-operator-fd8ccff4c-kqf2g"] Jul 09 15:36:25.716327 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:36:25.716291 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mvxqn\" (UniqueName: \"kubernetes.io/projected/f0a24581-6eee-4f3a-adad-476ed7659833-kube-api-access-mvxqn\") pod \"openshift-lws-operator-fd8ccff4c-kqf2g\" (UID: \"f0a24581-6eee-4f3a-adad-476ed7659833\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-kqf2g" Jul 09 15:36:25.716327 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:36:25.716330 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/f0a24581-6eee-4f3a-adad-476ed7659833-tmp\") pod \"openshift-lws-operator-fd8ccff4c-kqf2g\" (UID: \"f0a24581-6eee-4f3a-adad-476ed7659833\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-kqf2g" Jul 09 15:36:25.817329 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:36:25.817251 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mvxqn\" (UniqueName: \"kubernetes.io/projected/f0a24581-6eee-4f3a-adad-476ed7659833-kube-api-access-mvxqn\") pod \"openshift-lws-operator-fd8ccff4c-kqf2g\" (UID: \"f0a24581-6eee-4f3a-adad-476ed7659833\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-kqf2g" Jul 09 15:36:25.817329 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:36:25.817287 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/f0a24581-6eee-4f3a-adad-476ed7659833-tmp\") pod \"openshift-lws-operator-fd8ccff4c-kqf2g\" (UID: \"f0a24581-6eee-4f3a-adad-476ed7659833\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-kqf2g" Jul 09 15:36:25.817615 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:36:25.817600 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/f0a24581-6eee-4f3a-adad-476ed7659833-tmp\") pod \"openshift-lws-operator-fd8ccff4c-kqf2g\" (UID: \"f0a24581-6eee-4f3a-adad-476ed7659833\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-kqf2g" Jul 09 15:36:25.833654 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:36:25.833618 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mvxqn\" (UniqueName: \"kubernetes.io/projected/f0a24581-6eee-4f3a-adad-476ed7659833-kube-api-access-mvxqn\") pod \"openshift-lws-operator-fd8ccff4c-kqf2g\" (UID: \"f0a24581-6eee-4f3a-adad-476ed7659833\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-kqf2g" Jul 09 15:36:25.860671 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:36:25.860642 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-kqf2g" Jul 09 15:36:26.008176 ip-10-0-133-183 kubenswrapper[2573]: W0709 15:36:26.008146 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podf0a24581_6eee_4f3a_adad_476ed7659833.slice/crio-765b988ff3c2e3ccb6b84ab852d066485c06f0b38548127e1ff0506497a303c4 WatchSource:0}: Error finding container 765b988ff3c2e3ccb6b84ab852d066485c06f0b38548127e1ff0506497a303c4: Status 404 returned error can't find the container with id 765b988ff3c2e3ccb6b84ab852d066485c06f0b38548127e1ff0506497a303c4 Jul 09 15:36:26.008317 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:36:26.008261 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/openshift-lws-operator-fd8ccff4c-kqf2g"] Jul 09 15:36:26.009738 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:36:26.009720 2573 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 09 15:36:26.105344 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:36:26.105249 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-kqf2g" event={"ID":"f0a24581-6eee-4f3a-adad-476ed7659833","Type":"ContainerStarted","Data":"765b988ff3c2e3ccb6b84ab852d066485c06f0b38548127e1ff0506497a303c4"} Jul 09 15:36:29.121300 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:36:29.121262 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-kqf2g" event={"ID":"f0a24581-6eee-4f3a-adad-476ed7659833","Type":"ContainerStarted","Data":"4d26296609f4dd9e6f15fb0dc7b05faec13baaa66bbb699560612af81390a28a"} Jul 09 15:36:29.141416 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:36:29.141357 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-kqf2g" podStartSLOduration=1.123226153 podStartE2EDuration="4.141337302s" totalImagesPullingTime="3.018111149s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 15:36:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 15:36:29.139188804 +0000 UTC m=+310.860695697" watchObservedRunningTime="2026-07-09 15:36:29.141337302 +0000 UTC m=+310.862844164" Jul 09 15:36:48.972212 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:36:48.972176 2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-9bb557cf6-lmk4b"] Jul 09 15:36:48.975447 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:36:48.975427 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-9bb557cf6-lmk4b" Jul 09 15:36:48.978296 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:36:48.978268 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-webhook-cert\"" Jul 09 15:36:48.978296 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:36:48.978290 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-dockercfg-44wlf\"" Jul 09 15:36:48.978540 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:36:48.978374 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\"" Jul 09 15:36:48.979236 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:36:48.979219 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-service-cert\"" Jul 09 15:36:48.979366 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:36:48.979321 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\"" Jul 09 15:36:48.989974 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:36:48.989944 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-9bb557cf6-lmk4b"] Jul 09 15:36:49.024793 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:36:49.024757 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x55xv\" (UniqueName: \"kubernetes.io/projected/d038f9a0-b62d-4b40-bbb0-a758173d50c0-kube-api-access-x55xv\") pod \"opendatahub-operator-controller-manager-9bb557cf6-lmk4b\" (UID: \"d038f9a0-b62d-4b40-bbb0-a758173d50c0\") " pod="opendatahub/opendatahub-operator-controller-manager-9bb557cf6-lmk4b" Jul 09 15:36:49.024973 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:36:49.024811 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/d038f9a0-b62d-4b40-bbb0-a758173d50c0-webhook-cert\") pod \"opendatahub-operator-controller-manager-9bb557cf6-lmk4b\" (UID: \"d038f9a0-b62d-4b40-bbb0-a758173d50c0\") " pod="opendatahub/opendatahub-operator-controller-manager-9bb557cf6-lmk4b" Jul 09 15:36:49.024973 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:36:49.024894 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/d038f9a0-b62d-4b40-bbb0-a758173d50c0-manifests\") pod \"opendatahub-operator-controller-manager-9bb557cf6-lmk4b\" (UID: \"d038f9a0-b62d-4b40-bbb0-a758173d50c0\") " pod="opendatahub/opendatahub-operator-controller-manager-9bb557cf6-lmk4b" Jul 09 15:36:49.025063 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:36:49.024980 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/d038f9a0-b62d-4b40-bbb0-a758173d50c0-apiservice-cert\") pod \"opendatahub-operator-controller-manager-9bb557cf6-lmk4b\" (UID: \"d038f9a0-b62d-4b40-bbb0-a758173d50c0\") " pod="opendatahub/opendatahub-operator-controller-manager-9bb557cf6-lmk4b" Jul 09 15:36:49.025063 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:36:49.025037 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/d038f9a0-b62d-4b40-bbb0-a758173d50c0-tmp\") pod \"opendatahub-operator-controller-manager-9bb557cf6-lmk4b\" (UID: \"d038f9a0-b62d-4b40-bbb0-a758173d50c0\") " pod="opendatahub/opendatahub-operator-controller-manager-9bb557cf6-lmk4b" Jul 09 15:36:49.126406 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:36:49.126358 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/d038f9a0-b62d-4b40-bbb0-a758173d50c0-apiservice-cert\") pod \"opendatahub-operator-controller-manager-9bb557cf6-lmk4b\" (UID: \"d038f9a0-b62d-4b40-bbb0-a758173d50c0\") " pod="opendatahub/opendatahub-operator-controller-manager-9bb557cf6-lmk4b" Jul 09 15:36:49.126406 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:36:49.126410 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/d038f9a0-b62d-4b40-bbb0-a758173d50c0-tmp\") pod \"opendatahub-operator-controller-manager-9bb557cf6-lmk4b\" (UID: \"d038f9a0-b62d-4b40-bbb0-a758173d50c0\") " pod="opendatahub/opendatahub-operator-controller-manager-9bb557cf6-lmk4b" Jul 09 15:36:49.126632 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:36:49.126511 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x55xv\" (UniqueName: \"kubernetes.io/projected/d038f9a0-b62d-4b40-bbb0-a758173d50c0-kube-api-access-x55xv\") pod \"opendatahub-operator-controller-manager-9bb557cf6-lmk4b\" (UID: \"d038f9a0-b62d-4b40-bbb0-a758173d50c0\") " pod="opendatahub/opendatahub-operator-controller-manager-9bb557cf6-lmk4b" Jul 09 15:36:49.126632 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:36:49.126538 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/d038f9a0-b62d-4b40-bbb0-a758173d50c0-webhook-cert\") pod \"opendatahub-operator-controller-manager-9bb557cf6-lmk4b\" (UID: \"d038f9a0-b62d-4b40-bbb0-a758173d50c0\") " pod="opendatahub/opendatahub-operator-controller-manager-9bb557cf6-lmk4b" Jul 09 15:36:49.126632 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:36:49.126560 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/d038f9a0-b62d-4b40-bbb0-a758173d50c0-manifests\") pod \"opendatahub-operator-controller-manager-9bb557cf6-lmk4b\" (UID: \"d038f9a0-b62d-4b40-bbb0-a758173d50c0\") " pod="opendatahub/opendatahub-operator-controller-manager-9bb557cf6-lmk4b" Jul 09 15:36:49.126972 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:36:49.126947 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/d038f9a0-b62d-4b40-bbb0-a758173d50c0-manifests\") pod \"opendatahub-operator-controller-manager-9bb557cf6-lmk4b\" (UID: \"d038f9a0-b62d-4b40-bbb0-a758173d50c0\") " pod="opendatahub/opendatahub-operator-controller-manager-9bb557cf6-lmk4b" Jul 09 15:36:49.128820 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:36:49.128799 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/d038f9a0-b62d-4b40-bbb0-a758173d50c0-tmp\") pod \"opendatahub-operator-controller-manager-9bb557cf6-lmk4b\" (UID: \"d038f9a0-b62d-4b40-bbb0-a758173d50c0\") " pod="opendatahub/opendatahub-operator-controller-manager-9bb557cf6-lmk4b" Jul 09 15:36:49.128968 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:36:49.128952 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/d038f9a0-b62d-4b40-bbb0-a758173d50c0-webhook-cert\") pod \"opendatahub-operator-controller-manager-9bb557cf6-lmk4b\" (UID: \"d038f9a0-b62d-4b40-bbb0-a758173d50c0\") " pod="opendatahub/opendatahub-operator-controller-manager-9bb557cf6-lmk4b" Jul 09 15:36:49.129061 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:36:49.129044 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/d038f9a0-b62d-4b40-bbb0-a758173d50c0-apiservice-cert\") pod \"opendatahub-operator-controller-manager-9bb557cf6-lmk4b\" (UID: \"d038f9a0-b62d-4b40-bbb0-a758173d50c0\") " pod="opendatahub/opendatahub-operator-controller-manager-9bb557cf6-lmk4b" Jul 09 15:36:49.148442 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:36:49.148409 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-x55xv\" (UniqueName: \"kubernetes.io/projected/d038f9a0-b62d-4b40-bbb0-a758173d50c0-kube-api-access-x55xv\") pod \"opendatahub-operator-controller-manager-9bb557cf6-lmk4b\" (UID: \"d038f9a0-b62d-4b40-bbb0-a758173d50c0\") " pod="opendatahub/opendatahub-operator-controller-manager-9bb557cf6-lmk4b" Jul 09 15:36:49.285656 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:36:49.285565 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-9bb557cf6-lmk4b" Jul 09 15:36:49.459350 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:36:49.459315 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-9bb557cf6-lmk4b"] Jul 09 15:36:49.463829 ip-10-0-133-183 kubenswrapper[2573]: W0709 15:36:49.463804 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd038f9a0_b62d_4b40_bbb0_a758173d50c0.slice/crio-d35f346a183448326ecc3f9a4b1bf9f84dc5771cf647c5e04516e9df10437a6e WatchSource:0}: Error finding container d35f346a183448326ecc3f9a4b1bf9f84dc5771cf647c5e04516e9df10437a6e: Status 404 returned error can't find the container with id d35f346a183448326ecc3f9a4b1bf9f84dc5771cf647c5e04516e9df10437a6e Jul 09 15:36:50.210958 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:36:50.210914 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-9bb557cf6-lmk4b" event={"ID":"d038f9a0-b62d-4b40-bbb0-a758173d50c0","Type":"ContainerStarted","Data":"d35f346a183448326ecc3f9a4b1bf9f84dc5771cf647c5e04516e9df10437a6e"} Jul 09 15:36:53.226797 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:36:53.226760 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-9bb557cf6-lmk4b" event={"ID":"d038f9a0-b62d-4b40-bbb0-a758173d50c0","Type":"ContainerStarted","Data":"5366578aeaed6403000c195388e3eab6a564ba8695faffd8fc7a9674d8058649"} Jul 09 15:36:54.232308 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:36:54.232265 2573 generic.go:356] "Generic (PLEG): container finished" podID="d038f9a0-b62d-4b40-bbb0-a758173d50c0" containerID="5366578aeaed6403000c195388e3eab6a564ba8695faffd8fc7a9674d8058649" exitCode=0 Jul 09 15:36:54.232683 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:36:54.232311 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-9bb557cf6-lmk4b" event={"ID":"d038f9a0-b62d-4b40-bbb0-a758173d50c0","Type":"ContainerDied","Data":"5366578aeaed6403000c195388e3eab6a564ba8695faffd8fc7a9674d8058649"} Jul 09 15:36:55.238105 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:36:55.238071 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-9bb557cf6-lmk4b" event={"ID":"d038f9a0-b62d-4b40-bbb0-a758173d50c0","Type":"ContainerStarted","Data":"134c20530ffbd4d70bf9841cecd9a4bbc8e6149ddebb197d5f976929673a0b62"} Jul 09 15:36:55.238465 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:36:55.238191 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/opendatahub-operator-controller-manager-9bb557cf6-lmk4b" Jul 09 15:36:55.275768 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:36:55.275712 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/opendatahub-operator-controller-manager-9bb557cf6-lmk4b" podStartSLOduration=2.890085539 podStartE2EDuration="7.275695083s" totalImagesPullingTime="3.335931849s" totalInitContainerRuntime="1.049677695s" isStatefulPod=false podCreationTimestamp="2026-07-09 15:36:48 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 15:36:55.27462674 +0000 UTC m=+336.996133608" watchObservedRunningTime="2026-07-09 15:36:55.275695083 +0000 UTC m=+336.997201951" Jul 09 15:37:06.245297 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:37:06.245267 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/opendatahub-operator-controller-manager-9bb557cf6-lmk4b" Jul 09 15:37:32.535223 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:37:32.535191 2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-lws-operator/lws-controller-manager-5c8fc68c98-rzldz"] Jul 09 15:37:32.543547 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:37:32.543519 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-5c8fc68c98-rzldz" Jul 09 15:37:32.548160 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:37:32.548131 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"lws-controller-manager-dockercfg-cv98g\"" Jul 09 15:37:32.548346 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:37:32.548334 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"metrics-server-cert\"" Jul 09 15:37:32.548430 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:37:32.548412 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"webhook-server-cert\"" Jul 09 15:37:32.548492 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:37:32.548474 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"lws-manager-config\"" Jul 09 15:37:32.550679 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:37:32.550657 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-5c8fc68c98-rzldz"] Jul 09 15:37:32.606542 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:37:32.606513 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/0cbf8218-3798-438b-836c-7d5e99be8137-manager-config\") pod \"lws-controller-manager-5c8fc68c98-rzldz\" (UID: \"0cbf8218-3798-438b-836c-7d5e99be8137\") " pod="openshift-lws-operator/lws-controller-manager-5c8fc68c98-rzldz" Jul 09 15:37:32.606542 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:37:32.606544 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/0cbf8218-3798-438b-836c-7d5e99be8137-metrics-cert\") pod \"lws-controller-manager-5c8fc68c98-rzldz\" (UID: \"0cbf8218-3798-438b-836c-7d5e99be8137\") " pod="openshift-lws-operator/lws-controller-manager-5c8fc68c98-rzldz" Jul 09 15:37:32.606735 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:37:32.606609 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/0cbf8218-3798-438b-836c-7d5e99be8137-cert\") pod \"lws-controller-manager-5c8fc68c98-rzldz\" (UID: \"0cbf8218-3798-438b-836c-7d5e99be8137\") " pod="openshift-lws-operator/lws-controller-manager-5c8fc68c98-rzldz" Jul 09 15:37:32.606735 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:37:32.606641 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x76hx\" (UniqueName: \"kubernetes.io/projected/0cbf8218-3798-438b-836c-7d5e99be8137-kube-api-access-x76hx\") pod \"lws-controller-manager-5c8fc68c98-rzldz\" (UID: \"0cbf8218-3798-438b-836c-7d5e99be8137\") " pod="openshift-lws-operator/lws-controller-manager-5c8fc68c98-rzldz" Jul 09 15:37:32.707393 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:37:32.707355 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/0cbf8218-3798-438b-836c-7d5e99be8137-manager-config\") pod \"lws-controller-manager-5c8fc68c98-rzldz\" (UID: \"0cbf8218-3798-438b-836c-7d5e99be8137\") " pod="openshift-lws-operator/lws-controller-manager-5c8fc68c98-rzldz" Jul 09 15:37:32.707393 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:37:32.707394 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/0cbf8218-3798-438b-836c-7d5e99be8137-metrics-cert\") pod \"lws-controller-manager-5c8fc68c98-rzldz\" (UID: \"0cbf8218-3798-438b-836c-7d5e99be8137\") " pod="openshift-lws-operator/lws-controller-manager-5c8fc68c98-rzldz" Jul 09 15:37:32.707600 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:37:32.707469 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/0cbf8218-3798-438b-836c-7d5e99be8137-cert\") pod \"lws-controller-manager-5c8fc68c98-rzldz\" (UID: \"0cbf8218-3798-438b-836c-7d5e99be8137\") " pod="openshift-lws-operator/lws-controller-manager-5c8fc68c98-rzldz" Jul 09 15:37:32.707600 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:37:32.707524 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x76hx\" (UniqueName: \"kubernetes.io/projected/0cbf8218-3798-438b-836c-7d5e99be8137-kube-api-access-x76hx\") pod \"lws-controller-manager-5c8fc68c98-rzldz\" (UID: \"0cbf8218-3798-438b-836c-7d5e99be8137\") " pod="openshift-lws-operator/lws-controller-manager-5c8fc68c98-rzldz" Jul 09 15:37:32.708154 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:37:32.708128 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/0cbf8218-3798-438b-836c-7d5e99be8137-manager-config\") pod \"lws-controller-manager-5c8fc68c98-rzldz\" (UID: \"0cbf8218-3798-438b-836c-7d5e99be8137\") " pod="openshift-lws-operator/lws-controller-manager-5c8fc68c98-rzldz" Jul 09 15:37:32.709976 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:37:32.709949 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/0cbf8218-3798-438b-836c-7d5e99be8137-metrics-cert\") pod \"lws-controller-manager-5c8fc68c98-rzldz\" (UID: \"0cbf8218-3798-438b-836c-7d5e99be8137\") " pod="openshift-lws-operator/lws-controller-manager-5c8fc68c98-rzldz" Jul 09 15:37:32.709976 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:37:32.709973 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/0cbf8218-3798-438b-836c-7d5e99be8137-cert\") pod \"lws-controller-manager-5c8fc68c98-rzldz\" (UID: \"0cbf8218-3798-438b-836c-7d5e99be8137\") " pod="openshift-lws-operator/lws-controller-manager-5c8fc68c98-rzldz" Jul 09 15:37:32.717736 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:37:32.717711 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-x76hx\" (UniqueName: \"kubernetes.io/projected/0cbf8218-3798-438b-836c-7d5e99be8137-kube-api-access-x76hx\") pod \"lws-controller-manager-5c8fc68c98-rzldz\" (UID: \"0cbf8218-3798-438b-836c-7d5e99be8137\") " pod="openshift-lws-operator/lws-controller-manager-5c8fc68c98-rzldz" Jul 09 15:37:32.853300 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:37:32.853217 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-5c8fc68c98-rzldz" Jul 09 15:37:32.988151 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:37:32.988126 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-5c8fc68c98-rzldz"] Jul 09 15:37:32.990689 ip-10-0-133-183 kubenswrapper[2573]: W0709 15:37:32.990656 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0cbf8218_3798_438b_836c_7d5e99be8137.slice/crio-b24bca8f419fd5894a4e6c8b4ea44729ec4b276a4ac1836f4641c012d00f5e7c WatchSource:0}: Error finding container b24bca8f419fd5894a4e6c8b4ea44729ec4b276a4ac1836f4641c012d00f5e7c: Status 404 returned error can't find the container with id b24bca8f419fd5894a4e6c8b4ea44729ec4b276a4ac1836f4641c012d00f5e7c Jul 09 15:37:33.398946 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:37:33.398908 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-5c8fc68c98-rzldz" event={"ID":"0cbf8218-3798-438b-836c-7d5e99be8137","Type":"ContainerStarted","Data":"b24bca8f419fd5894a4e6c8b4ea44729ec4b276a4ac1836f4641c012d00f5e7c"} Jul 09 15:37:33.732363 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:37:33.732328 2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-94bb8fbfd-68ctz"] Jul 09 15:37:33.735376 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:37:33.735355 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-68ctz" Jul 09 15:37:33.740207 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:37:33.740186 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-tls\"" Jul 09 15:37:33.740326 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:37:33.740190 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\"" Jul 09 15:37:33.740387 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:37:33.740351 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-openshift-gateway-dockercfg-6qz8f\"" Jul 09 15:37:33.741487 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:37:33.741469 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\"" Jul 09 15:37:33.742458 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:37:33.742435 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\"" Jul 09 15:37:33.742646 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:37:33.742633 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"cacerts\"" Jul 09 15:37:33.743023 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:37:33.743008 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istio-kubeconfig\"" Jul 09 15:37:33.750598 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:37:33.750577 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-94bb8fbfd-68ctz"] Jul 09 15:37:33.816220 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:37:33.816185 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wf6zx\" (UniqueName: \"kubernetes.io/projected/10ae1018-c000-41b2-bd1c-e933573ae519-kube-api-access-wf6zx\") pod \"istiod-openshift-gateway-94bb8fbfd-68ctz\" (UID: \"10ae1018-c000-41b2-bd1c-e933573ae519\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-68ctz" Jul 09 15:37:33.816405 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:37:33.816240 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/10ae1018-c000-41b2-bd1c-e933573ae519-local-certs\") pod \"istiod-openshift-gateway-94bb8fbfd-68ctz\" (UID: \"10ae1018-c000-41b2-bd1c-e933573ae519\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-68ctz" Jul 09 15:37:33.816405 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:37:33.816361 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/10ae1018-c000-41b2-bd1c-e933573ae519-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-94bb8fbfd-68ctz\" (UID: \"10ae1018-c000-41b2-bd1c-e933573ae519\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-68ctz" Jul 09 15:37:33.816405 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:37:33.816400 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/10ae1018-c000-41b2-bd1c-e933573ae519-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-94bb8fbfd-68ctz\" (UID: \"10ae1018-c000-41b2-bd1c-e933573ae519\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-68ctz" Jul 09 15:37:33.816547 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:37:33.816462 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/10ae1018-c000-41b2-bd1c-e933573ae519-istio-token\") pod \"istiod-openshift-gateway-94bb8fbfd-68ctz\" (UID: \"10ae1018-c000-41b2-bd1c-e933573ae519\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-68ctz" Jul 09 15:37:33.816547 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:37:33.816495 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/10ae1018-c000-41b2-bd1c-e933573ae519-istio-kubeconfig\") pod \"istiod-openshift-gateway-94bb8fbfd-68ctz\" (UID: \"10ae1018-c000-41b2-bd1c-e933573ae519\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-68ctz" Jul 09 15:37:33.816547 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:37:33.816523 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/10ae1018-c000-41b2-bd1c-e933573ae519-cacerts\") pod \"istiod-openshift-gateway-94bb8fbfd-68ctz\" (UID: \"10ae1018-c000-41b2-bd1c-e933573ae519\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-68ctz" Jul 09 15:37:33.917561 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:37:33.917530 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wf6zx\" (UniqueName: \"kubernetes.io/projected/10ae1018-c000-41b2-bd1c-e933573ae519-kube-api-access-wf6zx\") pod \"istiod-openshift-gateway-94bb8fbfd-68ctz\" (UID: \"10ae1018-c000-41b2-bd1c-e933573ae519\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-68ctz" Jul 09 15:37:33.917561 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:37:33.917569 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/10ae1018-c000-41b2-bd1c-e933573ae519-local-certs\") pod \"istiod-openshift-gateway-94bb8fbfd-68ctz\" (UID: \"10ae1018-c000-41b2-bd1c-e933573ae519\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-68ctz" Jul 09 15:37:33.917817 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:37:33.917634 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/10ae1018-c000-41b2-bd1c-e933573ae519-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-94bb8fbfd-68ctz\" (UID: \"10ae1018-c000-41b2-bd1c-e933573ae519\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-68ctz" Jul 09 15:37:33.917817 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:37:33.917668 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/10ae1018-c000-41b2-bd1c-e933573ae519-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-94bb8fbfd-68ctz\" (UID: \"10ae1018-c000-41b2-bd1c-e933573ae519\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-68ctz" Jul 09 15:37:33.917817 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:37:33.917710 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/10ae1018-c000-41b2-bd1c-e933573ae519-istio-token\") pod \"istiod-openshift-gateway-94bb8fbfd-68ctz\" (UID: \"10ae1018-c000-41b2-bd1c-e933573ae519\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-68ctz" Jul 09 15:37:33.917817 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:37:33.917742 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/10ae1018-c000-41b2-bd1c-e933573ae519-istio-kubeconfig\") pod \"istiod-openshift-gateway-94bb8fbfd-68ctz\" (UID: \"10ae1018-c000-41b2-bd1c-e933573ae519\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-68ctz" Jul 09 15:37:33.917817 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:37:33.917768 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/10ae1018-c000-41b2-bd1c-e933573ae519-cacerts\") pod \"istiod-openshift-gateway-94bb8fbfd-68ctz\" (UID: \"10ae1018-c000-41b2-bd1c-e933573ae519\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-68ctz" Jul 09 15:37:33.918741 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:37:33.918712 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/10ae1018-c000-41b2-bd1c-e933573ae519-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-94bb8fbfd-68ctz\" (UID: \"10ae1018-c000-41b2-bd1c-e933573ae519\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-68ctz" Jul 09 15:37:33.920616 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:37:33.920298 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/10ae1018-c000-41b2-bd1c-e933573ae519-local-certs\") pod \"istiod-openshift-gateway-94bb8fbfd-68ctz\" (UID: \"10ae1018-c000-41b2-bd1c-e933573ae519\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-68ctz" Jul 09 15:37:33.920616 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:37:33.920569 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/10ae1018-c000-41b2-bd1c-e933573ae519-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-94bb8fbfd-68ctz\" (UID: \"10ae1018-c000-41b2-bd1c-e933573ae519\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-68ctz" Jul 09 15:37:33.920616 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:37:33.920615 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/10ae1018-c000-41b2-bd1c-e933573ae519-cacerts\") pod \"istiod-openshift-gateway-94bb8fbfd-68ctz\" (UID: \"10ae1018-c000-41b2-bd1c-e933573ae519\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-68ctz" Jul 09 15:37:33.920844 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:37:33.920807 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/10ae1018-c000-41b2-bd1c-e933573ae519-istio-kubeconfig\") pod \"istiod-openshift-gateway-94bb8fbfd-68ctz\" (UID: \"10ae1018-c000-41b2-bd1c-e933573ae519\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-68ctz" Jul 09 15:37:33.930739 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:37:33.930715 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wf6zx\" (UniqueName: \"kubernetes.io/projected/10ae1018-c000-41b2-bd1c-e933573ae519-kube-api-access-wf6zx\") pod \"istiod-openshift-gateway-94bb8fbfd-68ctz\" (UID: \"10ae1018-c000-41b2-bd1c-e933573ae519\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-68ctz" Jul 09 15:37:33.930828 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:37:33.930774 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/10ae1018-c000-41b2-bd1c-e933573ae519-istio-token\") pod \"istiod-openshift-gateway-94bb8fbfd-68ctz\" (UID: \"10ae1018-c000-41b2-bd1c-e933573ae519\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-68ctz" Jul 09 15:37:34.044842 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:37:34.044754 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-68ctz" Jul 09 15:37:34.202720 ip-10-0-133-183 kubenswrapper[2573]: W0709 15:37:34.202678 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod10ae1018_c000_41b2_bd1c_e933573ae519.slice/crio-f15218cc632f63c0a28609615ea1d197c00d13fa69cdc43a6d873a59ec66c035 WatchSource:0}: Error finding container f15218cc632f63c0a28609615ea1d197c00d13fa69cdc43a6d873a59ec66c035: Status 404 returned error can't find the container with id f15218cc632f63c0a28609615ea1d197c00d13fa69cdc43a6d873a59ec66c035 Jul 09 15:37:34.215367 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:37:34.215335 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-94bb8fbfd-68ctz"] Jul 09 15:37:34.407796 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:37:34.407716 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-68ctz" event={"ID":"10ae1018-c000-41b2-bd1c-e933573ae519","Type":"ContainerStarted","Data":"f15218cc632f63c0a28609615ea1d197c00d13fa69cdc43a6d873a59ec66c035"} Jul 09 15:37:36.419635 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:37:36.419602 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-5c8fc68c98-rzldz" event={"ID":"0cbf8218-3798-438b-836c-7d5e99be8137","Type":"ContainerStarted","Data":"e5efb3191e236335953f21adc18714aff0e52fe9352d1a9525b21d6bb9aac95d"} Jul 09 15:37:36.420121 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:37:36.419717 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-lws-operator/lws-controller-manager-5c8fc68c98-rzldz" Jul 09 15:37:36.544287 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:37:36.544237 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-lws-operator/lws-controller-manager-5c8fc68c98-rzldz" podStartSLOduration=2.195419627 podStartE2EDuration="4.54421957s" totalImagesPullingTime="2.348799943s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 15:37:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 15:37:36.543226641 +0000 UTC m=+378.264733510" watchObservedRunningTime="2026-07-09 15:37:36.54421957 +0000 UTC m=+378.265726438" Jul 09 15:37:36.571379 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:37:36.571333 2573 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 09 15:37:36.571517 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:37:36.571409 2573 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 09 15:37:37.426393 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:37:37.426355 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-68ctz" event={"ID":"10ae1018-c000-41b2-bd1c-e933573ae519","Type":"ContainerStarted","Data":"479553921d87b9b2d3f02a0105cb53c3d81a394c516f6a4621b94bd418c9e5fb"} Jul 09 15:37:37.479998 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:37:37.479933 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-68ctz" podStartSLOduration=2.113886306 podStartE2EDuration="4.479908858s" totalImagesPullingTime="2.366022552s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 15:37:33 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 15:37:37.47924853 +0000 UTC m=+379.200755413" watchObservedRunningTime="2026-07-09 15:37:37.479908858 +0000 UTC m=+379.201415727" Jul 09 15:37:38.430934 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:37:38.430904 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-68ctz" Jul 09 15:37:38.432365 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:37:38.432339 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-68ctz" Jul 09 15:37:47.428828 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:37:47.428798 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-lws-operator/lws-controller-manager-5c8fc68c98-rzldz" Jul 09 15:39:59.975039 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:39:59.975004 2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-gwj8q"] Jul 09 15:39:59.978063 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:39:59.978046 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-64c94fd64f-gwj8q" Jul 09 15:39:59.982625 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:39:59.982602 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"limitador-limits-config-limitador\"" Jul 09 15:39:59.982625 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:39:59.982627 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\"" Jul 09 15:39:59.982828 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:39:59.982693 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\"" Jul 09 15:39:59.984042 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:39:59.984016 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-ww2fh\"" Jul 09 15:39:59.991468 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:39:59.991447 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-gwj8q"] Jul 09 15:40:00.011023 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:40:00.010997 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x2tf4\" (UniqueName: \"kubernetes.io/projected/e7ef2616-66f9-40a6-b6a3-50d6076a861c-kube-api-access-x2tf4\") pod \"limitador-limitador-64c94fd64f-gwj8q\" (UID: \"e7ef2616-66f9-40a6-b6a3-50d6076a861c\") " pod="kuadrant-system/limitador-limitador-64c94fd64f-gwj8q" Jul 09 15:40:00.011160 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:40:00.011045 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/e7ef2616-66f9-40a6-b6a3-50d6076a861c-config-file\") pod \"limitador-limitador-64c94fd64f-gwj8q\" (UID: \"e7ef2616-66f9-40a6-b6a3-50d6076a861c\") " pod="kuadrant-system/limitador-limitador-64c94fd64f-gwj8q" Jul 09 15:40:00.076476 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:40:00.076439 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-gwj8q"] Jul 09 15:40:00.111908 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:40:00.111844 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x2tf4\" (UniqueName: \"kubernetes.io/projected/e7ef2616-66f9-40a6-b6a3-50d6076a861c-kube-api-access-x2tf4\") pod \"limitador-limitador-64c94fd64f-gwj8q\" (UID: \"e7ef2616-66f9-40a6-b6a3-50d6076a861c\") " pod="kuadrant-system/limitador-limitador-64c94fd64f-gwj8q" Jul 09 15:40:00.112088 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:40:00.112020 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/e7ef2616-66f9-40a6-b6a3-50d6076a861c-config-file\") pod \"limitador-limitador-64c94fd64f-gwj8q\" (UID: \"e7ef2616-66f9-40a6-b6a3-50d6076a861c\") " pod="kuadrant-system/limitador-limitador-64c94fd64f-gwj8q" Jul 09 15:40:00.112671 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:40:00.112647 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/e7ef2616-66f9-40a6-b6a3-50d6076a861c-config-file\") pod \"limitador-limitador-64c94fd64f-gwj8q\" (UID: \"e7ef2616-66f9-40a6-b6a3-50d6076a861c\") " pod="kuadrant-system/limitador-limitador-64c94fd64f-gwj8q" Jul 09 15:40:00.126707 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:40:00.126676 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-x2tf4\" (UniqueName: \"kubernetes.io/projected/e7ef2616-66f9-40a6-b6a3-50d6076a861c-kube-api-access-x2tf4\") pod \"limitador-limitador-64c94fd64f-gwj8q\" (UID: \"e7ef2616-66f9-40a6-b6a3-50d6076a861c\") " pod="kuadrant-system/limitador-limitador-64c94fd64f-gwj8q" Jul 09 15:40:00.289253 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:40:00.289148 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-64c94fd64f-gwj8q" Jul 09 15:40:00.452855 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:40:00.452728 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-gwj8q"] Jul 09 15:40:00.455619 ip-10-0-133-183 kubenswrapper[2573]: W0709 15:40:00.455588 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode7ef2616_66f9_40a6_b6a3_50d6076a861c.slice/crio-866579051a21d6c4cdd76435014f9b269f4506e0824050f8d83a696dd80428d5 WatchSource:0}: Error finding container 866579051a21d6c4cdd76435014f9b269f4506e0824050f8d83a696dd80428d5: Status 404 returned error can't find the container with id 866579051a21d6c4cdd76435014f9b269f4506e0824050f8d83a696dd80428d5 Jul 09 15:40:01.036911 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:40:01.036857 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-64c94fd64f-gwj8q" event={"ID":"e7ef2616-66f9-40a6-b6a3-50d6076a861c","Type":"ContainerStarted","Data":"866579051a21d6c4cdd76435014f9b269f4506e0824050f8d83a696dd80428d5"} Jul 09 15:40:04.059377 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:40:04.059341 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-64c94fd64f-gwj8q" event={"ID":"e7ef2616-66f9-40a6-b6a3-50d6076a861c","Type":"ContainerStarted","Data":"6738d8ca9a0f8b40b32ba2364e6d4ec85377879749d37ba02d56dad991357e33"} Jul 09 15:40:04.059785 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:40:04.059436 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-limitador-64c94fd64f-gwj8q" Jul 09 15:40:04.082568 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:40:04.082516 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-limitador-64c94fd64f-gwj8q" podStartSLOduration=1.631306538 podStartE2EDuration="5.082499518s" totalImagesPullingTime="3.45119298s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 15:39:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 15:40:04.080219841 +0000 UTC m=+525.801726711" watchObservedRunningTime="2026-07-09 15:40:04.082499518 +0000 UTC m=+525.804006386" Jul 09 15:40:15.065359 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:40:15.065280 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-limitador-64c94fd64f-gwj8q" Jul 09 15:40:15.835994 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:40:15.835955 2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-gwj8q"] Jul 09 15:40:15.836353 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:40:15.836137 2573 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/limitador-limitador-64c94fd64f-gwj8q" podUID="e7ef2616-66f9-40a6-b6a3-50d6076a861c" containerName="limitador" containerID="cri-o://6738d8ca9a0f8b40b32ba2364e6d4ec85377879749d37ba02d56dad991357e33" gracePeriod=30 Jul 09 15:40:16.387500 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:40:16.387474 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-64c94fd64f-gwj8q" Jul 09 15:40:16.455356 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:40:16.455331 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-x2tf4\" (UniqueName: \"kubernetes.io/projected/e7ef2616-66f9-40a6-b6a3-50d6076a861c-kube-api-access-x2tf4\") pod \"e7ef2616-66f9-40a6-b6a3-50d6076a861c\" (UID: \"e7ef2616-66f9-40a6-b6a3-50d6076a861c\") " Jul 09 15:40:16.455536 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:40:16.455425 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/e7ef2616-66f9-40a6-b6a3-50d6076a861c-config-file\") pod \"e7ef2616-66f9-40a6-b6a3-50d6076a861c\" (UID: \"e7ef2616-66f9-40a6-b6a3-50d6076a861c\") " Jul 09 15:40:16.455766 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:40:16.455741 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e7ef2616-66f9-40a6-b6a3-50d6076a861c-config-file" (OuterVolumeSpecName: "config-file") pod "e7ef2616-66f9-40a6-b6a3-50d6076a861c" (UID: "e7ef2616-66f9-40a6-b6a3-50d6076a861c"). InnerVolumeSpecName "config-file". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 09 15:40:16.457565 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:40:16.457546 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e7ef2616-66f9-40a6-b6a3-50d6076a861c-kube-api-access-x2tf4" (OuterVolumeSpecName: "kube-api-access-x2tf4") pod "e7ef2616-66f9-40a6-b6a3-50d6076a861c" (UID: "e7ef2616-66f9-40a6-b6a3-50d6076a861c"). InnerVolumeSpecName "kube-api-access-x2tf4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 09 15:40:16.556830 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:40:16.556790 2573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x2tf4\" (UniqueName: \"kubernetes.io/projected/e7ef2616-66f9-40a6-b6a3-50d6076a861c-kube-api-access-x2tf4\") on node \"ip-10-0-133-183.ec2.internal\" DevicePath \"\"" Jul 09 15:40:16.556830 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:40:16.556825 2573 reconciler_common.go:299] "Volume detached for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/e7ef2616-66f9-40a6-b6a3-50d6076a861c-config-file\") on node \"ip-10-0-133-183.ec2.internal\" DevicePath \"\"" Jul 09 15:40:17.119667 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:40:17.119580 2573 generic.go:356] "Generic (PLEG): container finished" podID="e7ef2616-66f9-40a6-b6a3-50d6076a861c" containerID="6738d8ca9a0f8b40b32ba2364e6d4ec85377879749d37ba02d56dad991357e33" exitCode=0 Jul 09 15:40:17.119667 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:40:17.119625 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-64c94fd64f-gwj8q" Jul 09 15:40:17.119667 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:40:17.119658 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-64c94fd64f-gwj8q" event={"ID":"e7ef2616-66f9-40a6-b6a3-50d6076a861c","Type":"ContainerDied","Data":"6738d8ca9a0f8b40b32ba2364e6d4ec85377879749d37ba02d56dad991357e33"} Jul 09 15:40:17.119927 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:40:17.119699 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-64c94fd64f-gwj8q" event={"ID":"e7ef2616-66f9-40a6-b6a3-50d6076a861c","Type":"ContainerDied","Data":"866579051a21d6c4cdd76435014f9b269f4506e0824050f8d83a696dd80428d5"} Jul 09 15:40:17.119927 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:40:17.119719 2573 scope.go:117] "RemoveContainer" containerID="6738d8ca9a0f8b40b32ba2364e6d4ec85377879749d37ba02d56dad991357e33" Jul 09 15:40:17.128089 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:40:17.128068 2573 scope.go:117] "RemoveContainer" containerID="6738d8ca9a0f8b40b32ba2364e6d4ec85377879749d37ba02d56dad991357e33" Jul 09 15:40:17.128353 ip-10-0-133-183 kubenswrapper[2573]: E0709 15:40:17.128332 2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6738d8ca9a0f8b40b32ba2364e6d4ec85377879749d37ba02d56dad991357e33\": container with ID starting with 6738d8ca9a0f8b40b32ba2364e6d4ec85377879749d37ba02d56dad991357e33 not found: ID does not exist" containerID="6738d8ca9a0f8b40b32ba2364e6d4ec85377879749d37ba02d56dad991357e33" Jul 09 15:40:17.128398 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:40:17.128363 2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6738d8ca9a0f8b40b32ba2364e6d4ec85377879749d37ba02d56dad991357e33"} err="failed to get container status \"6738d8ca9a0f8b40b32ba2364e6d4ec85377879749d37ba02d56dad991357e33\": rpc error: code = NotFound desc = could not find container \"6738d8ca9a0f8b40b32ba2364e6d4ec85377879749d37ba02d56dad991357e33\": container with ID starting with 6738d8ca9a0f8b40b32ba2364e6d4ec85377879749d37ba02d56dad991357e33 not found: ID does not exist" Jul 09 15:40:17.140955 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:40:17.140932 2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-gwj8q"] Jul 09 15:40:17.146249 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:40:17.146223 2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-gwj8q"] Jul 09 15:40:18.811536 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:40:18.811494 2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e7ef2616-66f9-40a6-b6a3-50d6076a861c" path="/var/lib/kubelet/pods/e7ef2616-66f9-40a6-b6a3-50d6076a861c/volumes" Jul 09 15:40:21.707922 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:40:21.707884 2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/postgres-69fb4594cf-xnzz7"] Jul 09 15:40:21.708429 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:40:21.708227 2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e7ef2616-66f9-40a6-b6a3-50d6076a861c" containerName="limitador" Jul 09 15:40:21.708429 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:40:21.708239 2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="e7ef2616-66f9-40a6-b6a3-50d6076a861c" containerName="limitador" Jul 09 15:40:21.708429 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:40:21.708290 2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="e7ef2616-66f9-40a6-b6a3-50d6076a861c" containerName="limitador" Jul 09 15:40:21.712773 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:40:21.712750 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/postgres-69fb4594cf-xnzz7" Jul 09 15:40:21.715693 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:40:21.715662 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"postgres-creds\"" Jul 09 15:40:21.715832 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:40:21.715689 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Jul 09 15:40:21.715832 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:40:21.715701 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Jul 09 15:40:21.716586 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:40:21.716570 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"default-dockercfg-vwgmj\"" Jul 09 15:40:21.720210 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:40:21.720184 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/postgres-69fb4594cf-xnzz7"] Jul 09 15:40:21.803283 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:40:21.803244 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ftp6v\" (UniqueName: \"kubernetes.io/projected/3dac9c89-baed-4cd0-97c0-5c7935d4915c-kube-api-access-ftp6v\") pod \"postgres-69fb4594cf-xnzz7\" (UID: \"3dac9c89-baed-4cd0-97c0-5c7935d4915c\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-xnzz7" Jul 09 15:40:21.803283 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:40:21.803285 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/3dac9c89-baed-4cd0-97c0-5c7935d4915c-data\") pod \"postgres-69fb4594cf-xnzz7\" (UID: \"3dac9c89-baed-4cd0-97c0-5c7935d4915c\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-xnzz7" Jul 09 15:40:21.904578 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:40:21.904542 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ftp6v\" (UniqueName: \"kubernetes.io/projected/3dac9c89-baed-4cd0-97c0-5c7935d4915c-kube-api-access-ftp6v\") pod \"postgres-69fb4594cf-xnzz7\" (UID: \"3dac9c89-baed-4cd0-97c0-5c7935d4915c\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-xnzz7" Jul 09 15:40:21.904578 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:40:21.904582 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/3dac9c89-baed-4cd0-97c0-5c7935d4915c-data\") pod \"postgres-69fb4594cf-xnzz7\" (UID: \"3dac9c89-baed-4cd0-97c0-5c7935d4915c\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-xnzz7" Jul 09 15:40:21.904997 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:40:21.904970 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/3dac9c89-baed-4cd0-97c0-5c7935d4915c-data\") pod \"postgres-69fb4594cf-xnzz7\" (UID: \"3dac9c89-baed-4cd0-97c0-5c7935d4915c\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-xnzz7" Jul 09 15:40:21.915489 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:40:21.915456 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ftp6v\" (UniqueName: \"kubernetes.io/projected/3dac9c89-baed-4cd0-97c0-5c7935d4915c-kube-api-access-ftp6v\") pod \"postgres-69fb4594cf-xnzz7\" (UID: \"3dac9c89-baed-4cd0-97c0-5c7935d4915c\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-xnzz7" Jul 09 15:40:22.024404 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:40:22.024327 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/postgres-69fb4594cf-xnzz7" Jul 09 15:40:22.165455 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:40:22.165431 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/postgres-69fb4594cf-xnzz7"] Jul 09 15:40:22.168192 ip-10-0-133-183 kubenswrapper[2573]: W0709 15:40:22.168163 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3dac9c89_baed_4cd0_97c0_5c7935d4915c.slice/crio-0825bc9502c3d485040fd2df770bdf12e97ef62c47d034dcafd43c648b894b0c WatchSource:0}: Error finding container 0825bc9502c3d485040fd2df770bdf12e97ef62c47d034dcafd43c648b894b0c: Status 404 returned error can't find the container with id 0825bc9502c3d485040fd2df770bdf12e97ef62c47d034dcafd43c648b894b0c Jul 09 15:40:23.149714 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:40:23.149662 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/postgres-69fb4594cf-xnzz7" event={"ID":"3dac9c89-baed-4cd0-97c0-5c7935d4915c","Type":"ContainerStarted","Data":"0825bc9502c3d485040fd2df770bdf12e97ef62c47d034dcafd43c648b894b0c"} Jul 09 15:40:27.173598 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:40:27.173554 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/postgres-69fb4594cf-xnzz7" event={"ID":"3dac9c89-baed-4cd0-97c0-5c7935d4915c","Type":"ContainerStarted","Data":"6922e70b8c05d602cf8a9c31627438d7199e5f1d4e9b22ec1886eedab3b75aa4"} Jul 09 15:40:27.174011 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:40:27.173682 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/postgres-69fb4594cf-xnzz7" Jul 09 15:40:27.196364 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:40:27.196296 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/postgres-69fb4594cf-xnzz7" podStartSLOduration=1.358039456 podStartE2EDuration="6.196270992s" totalImagesPullingTime="4.838231536s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 15:40:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 15:40:27.19281034 +0000 UTC m=+548.914317245" watchObservedRunningTime="2026-07-09 15:40:27.196270992 +0000 UTC m=+548.917777862" Jul 09 15:40:33.208658 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:40:33.208626 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/postgres-69fb4594cf-xnzz7" Jul 09 15:40:40.309096 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:40:40.309062 2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-5cb95ddc78-c85b4"] Jul 09 15:40:40.329218 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:40:40.329187 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-5cb95ddc78-c85b4"] Jul 09 15:40:40.329372 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:40:40.329282 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-5cb95ddc78-c85b4" Jul 09 15:40:40.334445 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:40:40.334416 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\"" Jul 09 15:40:40.334591 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:40:40.334473 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\"" Jul 09 15:40:40.334591 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:40:40.334498 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-vgq75\"" Jul 09 15:40:40.373425 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:40:40.373398 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fzxc8\" (UniqueName: \"kubernetes.io/projected/b8dd0424-bf45-4905-930a-605b7d77f804-kube-api-access-fzxc8\") pod \"maas-controller-5cb95ddc78-c85b4\" (UID: \"b8dd0424-bf45-4905-930a-605b7d77f804\") " pod="opendatahub/maas-controller-5cb95ddc78-c85b4" Jul 09 15:40:40.373583 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:40:40.373469 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/b8dd0424-bf45-4905-930a-605b7d77f804-webhook-cert\") pod \"maas-controller-5cb95ddc78-c85b4\" (UID: \"b8dd0424-bf45-4905-930a-605b7d77f804\") " pod="opendatahub/maas-controller-5cb95ddc78-c85b4" Jul 09 15:40:40.474210 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:40:40.474168 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/b8dd0424-bf45-4905-930a-605b7d77f804-webhook-cert\") pod \"maas-controller-5cb95ddc78-c85b4\" (UID: \"b8dd0424-bf45-4905-930a-605b7d77f804\") " pod="opendatahub/maas-controller-5cb95ddc78-c85b4" Jul 09 15:40:40.474377 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:40:40.474225 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fzxc8\" (UniqueName: \"kubernetes.io/projected/b8dd0424-bf45-4905-930a-605b7d77f804-kube-api-access-fzxc8\") pod \"maas-controller-5cb95ddc78-c85b4\" (UID: \"b8dd0424-bf45-4905-930a-605b7d77f804\") " pod="opendatahub/maas-controller-5cb95ddc78-c85b4" Jul 09 15:40:40.474377 ip-10-0-133-183 kubenswrapper[2573]: E0709 15:40:40.474317 2573 secret.go:189] Couldn't get secret opendatahub/maas-controller-webhook-cert: secret "maas-controller-webhook-cert" not found Jul 09 15:40:40.474446 ip-10-0-133-183 kubenswrapper[2573]: E0709 15:40:40.474379 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b8dd0424-bf45-4905-930a-605b7d77f804-webhook-cert podName:b8dd0424-bf45-4905-930a-605b7d77f804 nodeName:}" failed. No retries permitted until 2026-07-09 15:40:40.974362653 +0000 UTC m=+562.695869500 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "webhook-cert" (UniqueName: "kubernetes.io/secret/b8dd0424-bf45-4905-930a-605b7d77f804-webhook-cert") pod "maas-controller-5cb95ddc78-c85b4" (UID: "b8dd0424-bf45-4905-930a-605b7d77f804") : secret "maas-controller-webhook-cert" not found Jul 09 15:40:40.484507 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:40:40.484473 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fzxc8\" (UniqueName: \"kubernetes.io/projected/b8dd0424-bf45-4905-930a-605b7d77f804-kube-api-access-fzxc8\") pod \"maas-controller-5cb95ddc78-c85b4\" (UID: \"b8dd0424-bf45-4905-930a-605b7d77f804\") " pod="opendatahub/maas-controller-5cb95ddc78-c85b4" Jul 09 15:40:40.516607 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:40:40.516578 2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-647758b7f9-fgkbc"] Jul 09 15:40:40.519016 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:40:40.519001 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-647758b7f9-fgkbc" Jul 09 15:40:40.532302 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:40:40.532273 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-647758b7f9-fgkbc"] Jul 09 15:40:40.574655 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:40:40.574560 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5n92k\" (UniqueName: \"kubernetes.io/projected/47bb4e59-faa2-443d-85ce-711d6a4a157e-kube-api-access-5n92k\") pod \"maas-controller-647758b7f9-fgkbc\" (UID: \"47bb4e59-faa2-443d-85ce-711d6a4a157e\") " pod="opendatahub/maas-controller-647758b7f9-fgkbc" Jul 09 15:40:40.574655 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:40:40.574616 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/47bb4e59-faa2-443d-85ce-711d6a4a157e-webhook-cert\") pod \"maas-controller-647758b7f9-fgkbc\" (UID: \"47bb4e59-faa2-443d-85ce-711d6a4a157e\") " pod="opendatahub/maas-controller-647758b7f9-fgkbc" Jul 09 15:40:40.675146 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:40:40.675114 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5n92k\" (UniqueName: \"kubernetes.io/projected/47bb4e59-faa2-443d-85ce-711d6a4a157e-kube-api-access-5n92k\") pod \"maas-controller-647758b7f9-fgkbc\" (UID: \"47bb4e59-faa2-443d-85ce-711d6a4a157e\") " pod="opendatahub/maas-controller-647758b7f9-fgkbc" Jul 09 15:40:40.675329 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:40:40.675176 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/47bb4e59-faa2-443d-85ce-711d6a4a157e-webhook-cert\") pod \"maas-controller-647758b7f9-fgkbc\" (UID: \"47bb4e59-faa2-443d-85ce-711d6a4a157e\") " pod="opendatahub/maas-controller-647758b7f9-fgkbc" Jul 09 15:40:40.677476 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:40:40.677450 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/47bb4e59-faa2-443d-85ce-711d6a4a157e-webhook-cert\") pod \"maas-controller-647758b7f9-fgkbc\" (UID: \"47bb4e59-faa2-443d-85ce-711d6a4a157e\") " pod="opendatahub/maas-controller-647758b7f9-fgkbc" Jul 09 15:40:40.685239 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:40:40.685215 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5n92k\" (UniqueName: \"kubernetes.io/projected/47bb4e59-faa2-443d-85ce-711d6a4a157e-kube-api-access-5n92k\") pod \"maas-controller-647758b7f9-fgkbc\" (UID: \"47bb4e59-faa2-443d-85ce-711d6a4a157e\") " pod="opendatahub/maas-controller-647758b7f9-fgkbc" Jul 09 15:40:40.829840 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:40:40.829748 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-647758b7f9-fgkbc" Jul 09 15:40:40.963789 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:40:40.963764 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-647758b7f9-fgkbc"] Jul 09 15:40:40.966197 ip-10-0-133-183 kubenswrapper[2573]: W0709 15:40:40.966170 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod47bb4e59_faa2_443d_85ce_711d6a4a157e.slice/crio-fc6daff865690746506dba360afe23cea9d5f052150f82ed275129cf599d8e86 WatchSource:0}: Error finding container fc6daff865690746506dba360afe23cea9d5f052150f82ed275129cf599d8e86: Status 404 returned error can't find the container with id fc6daff865690746506dba360afe23cea9d5f052150f82ed275129cf599d8e86 Jul 09 15:40:40.977603 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:40:40.977578 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/b8dd0424-bf45-4905-930a-605b7d77f804-webhook-cert\") pod \"maas-controller-5cb95ddc78-c85b4\" (UID: \"b8dd0424-bf45-4905-930a-605b7d77f804\") " pod="opendatahub/maas-controller-5cb95ddc78-c85b4" Jul 09 15:40:40.979908 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:40:40.979863 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/b8dd0424-bf45-4905-930a-605b7d77f804-webhook-cert\") pod \"maas-controller-5cb95ddc78-c85b4\" (UID: \"b8dd0424-bf45-4905-930a-605b7d77f804\") " pod="opendatahub/maas-controller-5cb95ddc78-c85b4" Jul 09 15:40:41.234486 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:40:41.234440 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-647758b7f9-fgkbc" event={"ID":"47bb4e59-faa2-443d-85ce-711d6a4a157e","Type":"ContainerStarted","Data":"fc6daff865690746506dba360afe23cea9d5f052150f82ed275129cf599d8e86"} Jul 09 15:40:41.239647 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:40:41.239623 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-5cb95ddc78-c85b4" Jul 09 15:40:41.374257 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:40:41.374071 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-5cb95ddc78-c85b4"] Jul 09 15:40:41.376812 ip-10-0-133-183 kubenswrapper[2573]: W0709 15:40:41.376780 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb8dd0424_bf45_4905_930a_605b7d77f804.slice/crio-e9fc34cf32dc0315e9426b1cb4cf76f4b9b97bfc54a758030d3eaf081be9aa85 WatchSource:0}: Error finding container e9fc34cf32dc0315e9426b1cb4cf76f4b9b97bfc54a758030d3eaf081be9aa85: Status 404 returned error can't find the container with id e9fc34cf32dc0315e9426b1cb4cf76f4b9b97bfc54a758030d3eaf081be9aa85 Jul 09 15:40:42.240253 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:40:42.240215 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-5cb95ddc78-c85b4" event={"ID":"b8dd0424-bf45-4905-930a-605b7d77f804","Type":"ContainerStarted","Data":"e9fc34cf32dc0315e9426b1cb4cf76f4b9b97bfc54a758030d3eaf081be9aa85"} Jul 09 15:40:45.257154 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:40:45.257115 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-5cb95ddc78-c85b4" event={"ID":"b8dd0424-bf45-4905-930a-605b7d77f804","Type":"ContainerStarted","Data":"270e25f43c17477b0826336527ba5c77cd10205056c211e31504936c1f02b706"} Jul 09 15:40:45.257627 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:40:45.257191 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-5cb95ddc78-c85b4" Jul 09 15:40:45.258510 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:40:45.258490 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-647758b7f9-fgkbc" event={"ID":"47bb4e59-faa2-443d-85ce-711d6a4a157e","Type":"ContainerStarted","Data":"cd9094c792de52ac6225be10dbaf6e0ef55992f462a4c89141462fdeccc5641f"} Jul 09 15:40:45.258560 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:40:45.258536 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-647758b7f9-fgkbc" Jul 09 15:40:45.278168 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:40:45.278121 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-controller-5cb95ddc78-c85b4" podStartSLOduration=2.089262783 podStartE2EDuration="5.278104478s" totalImagesPullingTime="3.188841695s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 15:40:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 15:40:45.277092026 +0000 UTC m=+566.998598896" watchObservedRunningTime="2026-07-09 15:40:45.278104478 +0000 UTC m=+566.999611352" Jul 09 15:40:45.301504 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:40:45.301450 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-controller-647758b7f9-fgkbc" podStartSLOduration=1.695258673 podStartE2EDuration="5.301433061s" totalImagesPullingTime="3.606174388s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 15:40:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 15:40:45.299824358 +0000 UTC m=+567.021331227" watchObservedRunningTime="2026-07-09 15:40:45.301433061 +0000 UTC m=+567.022939987" Jul 09 15:40:56.268415 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:40:56.268383 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-5cb95ddc78-c85b4" Jul 09 15:40:56.268923 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:40:56.268460 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-647758b7f9-fgkbc" Jul 09 15:40:56.347120 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:40:56.347085 2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-5cb95ddc78-c85b4"] Jul 09 15:40:56.347362 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:40:56.347314 2573 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/maas-controller-5cb95ddc78-c85b4" podUID="b8dd0424-bf45-4905-930a-605b7d77f804" containerName="manager" containerID="cri-o://270e25f43c17477b0826336527ba5c77cd10205056c211e31504936c1f02b706" gracePeriod=30 Jul 09 15:40:56.589074 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:40:56.589048 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-5cb95ddc78-c85b4" Jul 09 15:40:56.726310 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:40:56.726279 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-fzxc8\" (UniqueName: \"kubernetes.io/projected/b8dd0424-bf45-4905-930a-605b7d77f804-kube-api-access-fzxc8\") pod \"b8dd0424-bf45-4905-930a-605b7d77f804\" (UID: \"b8dd0424-bf45-4905-930a-605b7d77f804\") " Jul 09 15:40:56.726486 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:40:56.726361 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/b8dd0424-bf45-4905-930a-605b7d77f804-webhook-cert\") pod \"b8dd0424-bf45-4905-930a-605b7d77f804\" (UID: \"b8dd0424-bf45-4905-930a-605b7d77f804\") " Jul 09 15:40:56.728554 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:40:56.728523 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b8dd0424-bf45-4905-930a-605b7d77f804-kube-api-access-fzxc8" (OuterVolumeSpecName: "kube-api-access-fzxc8") pod "b8dd0424-bf45-4905-930a-605b7d77f804" (UID: "b8dd0424-bf45-4905-930a-605b7d77f804"). InnerVolumeSpecName "kube-api-access-fzxc8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 09 15:40:56.728554 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:40:56.728538 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b8dd0424-bf45-4905-930a-605b7d77f804-webhook-cert" (OuterVolumeSpecName: "webhook-cert") pod "b8dd0424-bf45-4905-930a-605b7d77f804" (UID: "b8dd0424-bf45-4905-930a-605b7d77f804"). InnerVolumeSpecName "webhook-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 09 15:40:56.826937 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:40:56.826863 2573 reconciler_common.go:299] "Volume detached for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/b8dd0424-bf45-4905-930a-605b7d77f804-webhook-cert\") on node \"ip-10-0-133-183.ec2.internal\" DevicePath \"\"" Jul 09 15:40:56.826937 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:40:56.826898 2573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fzxc8\" (UniqueName: \"kubernetes.io/projected/b8dd0424-bf45-4905-930a-605b7d77f804-kube-api-access-fzxc8\") on node \"ip-10-0-133-183.ec2.internal\" DevicePath \"\"" Jul 09 15:40:57.314419 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:40:57.314379 2573 generic.go:356] "Generic (PLEG): container finished" podID="b8dd0424-bf45-4905-930a-605b7d77f804" containerID="270e25f43c17477b0826336527ba5c77cd10205056c211e31504936c1f02b706" exitCode=0 Jul 09 15:40:57.314811 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:40:57.314427 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-5cb95ddc78-c85b4" Jul 09 15:40:57.314811 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:40:57.314427 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-5cb95ddc78-c85b4" event={"ID":"b8dd0424-bf45-4905-930a-605b7d77f804","Type":"ContainerDied","Data":"270e25f43c17477b0826336527ba5c77cd10205056c211e31504936c1f02b706"} Jul 09 15:40:57.314811 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:40:57.314566 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-5cb95ddc78-c85b4" event={"ID":"b8dd0424-bf45-4905-930a-605b7d77f804","Type":"ContainerDied","Data":"e9fc34cf32dc0315e9426b1cb4cf76f4b9b97bfc54a758030d3eaf081be9aa85"} Jul 09 15:40:57.314811 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:40:57.314597 2573 scope.go:117] "RemoveContainer" containerID="270e25f43c17477b0826336527ba5c77cd10205056c211e31504936c1f02b706" Jul 09 15:40:57.323627 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:40:57.323607 2573 scope.go:117] "RemoveContainer" containerID="270e25f43c17477b0826336527ba5c77cd10205056c211e31504936c1f02b706" Jul 09 15:40:57.323998 ip-10-0-133-183 kubenswrapper[2573]: E0709 15:40:57.323970 2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"270e25f43c17477b0826336527ba5c77cd10205056c211e31504936c1f02b706\": container with ID starting with 270e25f43c17477b0826336527ba5c77cd10205056c211e31504936c1f02b706 not found: ID does not exist" containerID="270e25f43c17477b0826336527ba5c77cd10205056c211e31504936c1f02b706" Jul 09 15:40:57.324101 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:40:57.324008 2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"270e25f43c17477b0826336527ba5c77cd10205056c211e31504936c1f02b706"} err="failed to get container status \"270e25f43c17477b0826336527ba5c77cd10205056c211e31504936c1f02b706\": rpc error: code = NotFound desc = could not find container \"270e25f43c17477b0826336527ba5c77cd10205056c211e31504936c1f02b706\": container with ID starting with 270e25f43c17477b0826336527ba5c77cd10205056c211e31504936c1f02b706 not found: ID does not exist" Jul 09 15:40:57.336336 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:40:57.336310 2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-5cb95ddc78-c85b4"] Jul 09 15:40:57.343035 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:40:57.343011 2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-controller-5cb95ddc78-c85b4"] Jul 09 15:40:58.810071 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:40:58.810038 2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b8dd0424-bf45-4905-930a-605b7d77f804" path="/var/lib/kubelet/pods/b8dd0424-bf45-4905-930a-605b7d77f804/volumes" Jul 09 15:41:18.299433 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:41:18.299392 2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-6c45ffb8d8-6x46q"] Jul 09 15:41:18.300733 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:41:18.300705 2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b8dd0424-bf45-4905-930a-605b7d77f804" containerName="manager" Jul 09 15:41:18.300897 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:41:18.300866 2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="b8dd0424-bf45-4905-930a-605b7d77f804" containerName="manager" Jul 09 15:41:18.301150 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:41:18.301136 2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="b8dd0424-bf45-4905-930a-605b7d77f804" containerName="manager" Jul 09 15:41:18.306471 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:41:18.306444 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-6c45ffb8d8-6x46q" Jul 09 15:41:18.309550 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:41:18.309530 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-serving-cert\"" Jul 09 15:41:18.309550 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:41:18.309543 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-8jtfq\"" Jul 09 15:41:18.314940 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:41:18.314915 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-6c45ffb8d8-6x46q"] Jul 09 15:41:18.411281 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:41:18.411249 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/05ef5adf-5359-4075-a96b-6a8ae3f648b7-openshift-service-ca\") pod \"maas-api-6c45ffb8d8-6x46q\" (UID: \"05ef5adf-5359-4075-a96b-6a8ae3f648b7\") " pod="odh-ai-gateway-infra/maas-api-6c45ffb8d8-6x46q" Jul 09 15:41:18.411475 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:41:18.411301 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/05ef5adf-5359-4075-a96b-6a8ae3f648b7-maas-api-tls\") pod \"maas-api-6c45ffb8d8-6x46q\" (UID: \"05ef5adf-5359-4075-a96b-6a8ae3f648b7\") " pod="odh-ai-gateway-infra/maas-api-6c45ffb8d8-6x46q" Jul 09 15:41:18.411475 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:41:18.411337 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s9vhs\" (UniqueName: \"kubernetes.io/projected/05ef5adf-5359-4075-a96b-6a8ae3f648b7-kube-api-access-s9vhs\") pod \"maas-api-6c45ffb8d8-6x46q\" (UID: \"05ef5adf-5359-4075-a96b-6a8ae3f648b7\") " pod="odh-ai-gateway-infra/maas-api-6c45ffb8d8-6x46q" Jul 09 15:41:18.512393 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:41:18.512355 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/05ef5adf-5359-4075-a96b-6a8ae3f648b7-maas-api-tls\") pod \"maas-api-6c45ffb8d8-6x46q\" (UID: \"05ef5adf-5359-4075-a96b-6a8ae3f648b7\") " pod="odh-ai-gateway-infra/maas-api-6c45ffb8d8-6x46q" Jul 09 15:41:18.512393 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:41:18.512402 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s9vhs\" (UniqueName: \"kubernetes.io/projected/05ef5adf-5359-4075-a96b-6a8ae3f648b7-kube-api-access-s9vhs\") pod \"maas-api-6c45ffb8d8-6x46q\" (UID: \"05ef5adf-5359-4075-a96b-6a8ae3f648b7\") " pod="odh-ai-gateway-infra/maas-api-6c45ffb8d8-6x46q" Jul 09 15:41:18.512721 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:41:18.512481 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/05ef5adf-5359-4075-a96b-6a8ae3f648b7-openshift-service-ca\") pod \"maas-api-6c45ffb8d8-6x46q\" (UID: \"05ef5adf-5359-4075-a96b-6a8ae3f648b7\") " pod="odh-ai-gateway-infra/maas-api-6c45ffb8d8-6x46q" Jul 09 15:41:18.513173 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:41:18.513153 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/05ef5adf-5359-4075-a96b-6a8ae3f648b7-openshift-service-ca\") pod \"maas-api-6c45ffb8d8-6x46q\" (UID: \"05ef5adf-5359-4075-a96b-6a8ae3f648b7\") " pod="odh-ai-gateway-infra/maas-api-6c45ffb8d8-6x46q" Jul 09 15:41:18.514818 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:41:18.514793 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/05ef5adf-5359-4075-a96b-6a8ae3f648b7-maas-api-tls\") pod \"maas-api-6c45ffb8d8-6x46q\" (UID: \"05ef5adf-5359-4075-a96b-6a8ae3f648b7\") " pod="odh-ai-gateway-infra/maas-api-6c45ffb8d8-6x46q" Jul 09 15:41:18.522549 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:41:18.522522 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-s9vhs\" (UniqueName: \"kubernetes.io/projected/05ef5adf-5359-4075-a96b-6a8ae3f648b7-kube-api-access-s9vhs\") pod \"maas-api-6c45ffb8d8-6x46q\" (UID: \"05ef5adf-5359-4075-a96b-6a8ae3f648b7\") " pod="odh-ai-gateway-infra/maas-api-6c45ffb8d8-6x46q" Jul 09 15:41:18.621087 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:41:18.621010 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-8jtfq\"" Jul 09 15:41:18.628969 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:41:18.628935 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-6c45ffb8d8-6x46q" Jul 09 15:41:18.685223 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:41:18.685192 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-b7b9v_db22f88d-a953-488e-b384-67f991351f73/ovn-acl-logging/0.log" Jul 09 15:41:18.686753 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:41:18.686729 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-b7b9v_db22f88d-a953-488e-b384-67f991351f73/ovn-acl-logging/0.log" Jul 09 15:41:18.787201 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:41:18.787170 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-6c45ffb8d8-6x46q"] Jul 09 15:41:18.791652 ip-10-0-133-183 kubenswrapper[2573]: W0709 15:41:18.791624 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod05ef5adf_5359_4075_a96b_6a8ae3f648b7.slice/crio-f14ccc55efca0eecb1ae14a67c95c312961fe73bface1501f860dd1f24df69f9 WatchSource:0}: Error finding container f14ccc55efca0eecb1ae14a67c95c312961fe73bface1501f860dd1f24df69f9: Status 404 returned error can't find the container with id f14ccc55efca0eecb1ae14a67c95c312961fe73bface1501f860dd1f24df69f9 Jul 09 15:41:18.910719 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:41:18.910643 2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-58f86c7798-jkj5d"] Jul 09 15:41:18.913996 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:41:18.913972 2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-7964457696-m5gbg"] Jul 09 15:41:18.914177 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:41:18.914096 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-58f86c7798-jkj5d" Jul 09 15:41:18.916075 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:41:18.916057 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-7964457696-m5gbg" Jul 09 15:41:18.916857 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:41:18.916842 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins\"" Jul 09 15:41:18.918514 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:41:18.918499 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-dockercfg-8vfl6\"" Jul 09 15:41:18.925272 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:41:18.925248 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-58f86c7798-jkj5d"] Jul 09 15:41:18.939577 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:41:18.939551 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-7964457696-m5gbg"] Jul 09 15:41:19.015974 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:41:19.015931 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-84nrx\" (UniqueName: \"kubernetes.io/projected/c99251e1-0ac9-4fe6-9acf-e099c6bcafdc-kube-api-access-84nrx\") pod \"payload-processing-58f86c7798-jkj5d\" (UID: \"c99251e1-0ac9-4fe6-9acf-e099c6bcafdc\") " pod="openshift-ingress/payload-processing-58f86c7798-jkj5d" Jul 09 15:41:19.015974 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:41:19.015992 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-srn4f\" (UniqueName: \"kubernetes.io/projected/c7bca669-f5f2-41b2-87be-4ff0a7e25993-kube-api-access-srn4f\") pod \"payload-pre-processing-7964457696-m5gbg\" (UID: \"c7bca669-f5f2-41b2-87be-4ff0a7e25993\") " pod="openshift-ingress/payload-pre-processing-7964457696-m5gbg" Jul 09 15:41:19.016192 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:41:19.016023 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c99251e1-0ac9-4fe6-9acf-e099c6bcafdc-plugins-config-volume\") pod \"payload-processing-58f86c7798-jkj5d\" (UID: \"c99251e1-0ac9-4fe6-9acf-e099c6bcafdc\") " pod="openshift-ingress/payload-processing-58f86c7798-jkj5d" Jul 09 15:41:19.016192 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:41:19.016043 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c7bca669-f5f2-41b2-87be-4ff0a7e25993-plugins-config-volume\") pod \"payload-pre-processing-7964457696-m5gbg\" (UID: \"c7bca669-f5f2-41b2-87be-4ff0a7e25993\") " pod="openshift-ingress/payload-pre-processing-7964457696-m5gbg" Jul 09 15:41:19.116644 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:41:19.116612 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-84nrx\" (UniqueName: \"kubernetes.io/projected/c99251e1-0ac9-4fe6-9acf-e099c6bcafdc-kube-api-access-84nrx\") pod \"payload-processing-58f86c7798-jkj5d\" (UID: \"c99251e1-0ac9-4fe6-9acf-e099c6bcafdc\") " pod="openshift-ingress/payload-processing-58f86c7798-jkj5d" Jul 09 15:41:19.116836 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:41:19.116669 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-srn4f\" (UniqueName: \"kubernetes.io/projected/c7bca669-f5f2-41b2-87be-4ff0a7e25993-kube-api-access-srn4f\") pod \"payload-pre-processing-7964457696-m5gbg\" (UID: \"c7bca669-f5f2-41b2-87be-4ff0a7e25993\") " pod="openshift-ingress/payload-pre-processing-7964457696-m5gbg" Jul 09 15:41:19.116836 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:41:19.116706 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c99251e1-0ac9-4fe6-9acf-e099c6bcafdc-plugins-config-volume\") pod \"payload-processing-58f86c7798-jkj5d\" (UID: \"c99251e1-0ac9-4fe6-9acf-e099c6bcafdc\") " pod="openshift-ingress/payload-processing-58f86c7798-jkj5d" Jul 09 15:41:19.116836 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:41:19.116730 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c7bca669-f5f2-41b2-87be-4ff0a7e25993-plugins-config-volume\") pod \"payload-pre-processing-7964457696-m5gbg\" (UID: \"c7bca669-f5f2-41b2-87be-4ff0a7e25993\") " pod="openshift-ingress/payload-pre-processing-7964457696-m5gbg" Jul 09 15:41:19.117461 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:41:19.117443 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c7bca669-f5f2-41b2-87be-4ff0a7e25993-plugins-config-volume\") pod \"payload-pre-processing-7964457696-m5gbg\" (UID: \"c7bca669-f5f2-41b2-87be-4ff0a7e25993\") " pod="openshift-ingress/payload-pre-processing-7964457696-m5gbg" Jul 09 15:41:19.117509 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:41:19.117457 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c99251e1-0ac9-4fe6-9acf-e099c6bcafdc-plugins-config-volume\") pod \"payload-processing-58f86c7798-jkj5d\" (UID: \"c99251e1-0ac9-4fe6-9acf-e099c6bcafdc\") " pod="openshift-ingress/payload-processing-58f86c7798-jkj5d" Jul 09 15:41:19.128086 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:41:19.128060 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-84nrx\" (UniqueName: \"kubernetes.io/projected/c99251e1-0ac9-4fe6-9acf-e099c6bcafdc-kube-api-access-84nrx\") pod \"payload-processing-58f86c7798-jkj5d\" (UID: \"c99251e1-0ac9-4fe6-9acf-e099c6bcafdc\") " pod="openshift-ingress/payload-processing-58f86c7798-jkj5d" Jul 09 15:41:19.128209 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:41:19.128093 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-srn4f\" (UniqueName: \"kubernetes.io/projected/c7bca669-f5f2-41b2-87be-4ff0a7e25993-kube-api-access-srn4f\") pod \"payload-pre-processing-7964457696-m5gbg\" (UID: \"c7bca669-f5f2-41b2-87be-4ff0a7e25993\") " pod="openshift-ingress/payload-pre-processing-7964457696-m5gbg" Jul 09 15:41:19.227204 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:41:19.227168 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-58f86c7798-jkj5d" Jul 09 15:41:19.234192 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:41:19.234166 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-7964457696-m5gbg" Jul 09 15:41:19.384942 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:41:19.384871 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-58f86c7798-jkj5d"] Jul 09 15:41:19.390141 ip-10-0-133-183 kubenswrapper[2573]: W0709 15:41:19.390117 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc99251e1_0ac9_4fe6_9acf_e099c6bcafdc.slice/crio-26de5a8f1b2990df5e2e8b6500027aea5c71b509ad26898d5a9044ffba39920b WatchSource:0}: Error finding container 26de5a8f1b2990df5e2e8b6500027aea5c71b509ad26898d5a9044ffba39920b: Status 404 returned error can't find the container with id 26de5a8f1b2990df5e2e8b6500027aea5c71b509ad26898d5a9044ffba39920b Jul 09 15:41:19.402055 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:41:19.402032 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-7964457696-m5gbg"] Jul 09 15:41:19.402811 ip-10-0-133-183 kubenswrapper[2573]: W0709 15:41:19.402771 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc7bca669_f5f2_41b2_87be_4ff0a7e25993.slice/crio-30fb1f42b0234b944cb53583451b172da23d0e22c865a206921183afea7a29d9 WatchSource:0}: Error finding container 30fb1f42b0234b944cb53583451b172da23d0e22c865a206921183afea7a29d9: Status 404 returned error can't find the container with id 30fb1f42b0234b944cb53583451b172da23d0e22c865a206921183afea7a29d9 Jul 09 15:41:19.412357 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:41:19.412322 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-7964457696-m5gbg" event={"ID":"c7bca669-f5f2-41b2-87be-4ff0a7e25993","Type":"ContainerStarted","Data":"30fb1f42b0234b944cb53583451b172da23d0e22c865a206921183afea7a29d9"} Jul 09 15:41:19.413423 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:41:19.413398 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-6c45ffb8d8-6x46q" event={"ID":"05ef5adf-5359-4075-a96b-6a8ae3f648b7","Type":"ContainerStarted","Data":"f14ccc55efca0eecb1ae14a67c95c312961fe73bface1501f860dd1f24df69f9"} Jul 09 15:41:19.414544 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:41:19.414525 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-58f86c7798-jkj5d" event={"ID":"c99251e1-0ac9-4fe6-9acf-e099c6bcafdc","Type":"ContainerStarted","Data":"26de5a8f1b2990df5e2e8b6500027aea5c71b509ad26898d5a9044ffba39920b"} Jul 09 15:41:21.425413 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:41:21.425376 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-7964457696-m5gbg" event={"ID":"c7bca669-f5f2-41b2-87be-4ff0a7e25993","Type":"ContainerStarted","Data":"657d054f1a40ce32c9ffb7856824d73c64b7254909828b4187cc6574a178277a"} Jul 09 15:41:21.425796 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:41:21.425500 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-7964457696-m5gbg" Jul 09 15:41:21.427038 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:41:21.427014 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-6c45ffb8d8-6x46q" event={"ID":"05ef5adf-5359-4075-a96b-6a8ae3f648b7","Type":"ContainerStarted","Data":"e92d4686b561054176141cc77b9e4ed96ec3becfc56d0f392649d3ee9ac10188"} Jul 09 15:41:21.427153 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:41:21.427057 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-6c45ffb8d8-6x46q" Jul 09 15:41:21.428475 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:41:21.428451 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-58f86c7798-jkj5d" event={"ID":"c99251e1-0ac9-4fe6-9acf-e099c6bcafdc","Type":"ContainerStarted","Data":"4110a89c318694fdef99c56edebc6a36fc0c4c49242f443e8ae0476ef2417a46"} Jul 09 15:41:21.428584 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:41:21.428502 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-58f86c7798-jkj5d" Jul 09 15:41:21.446198 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:41:21.446155 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-7964457696-m5gbg" podStartSLOduration=1.56431455 podStartE2EDuration="3.44613982s" totalImagesPullingTime="1.88182527s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 15:41:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 15:41:21.444861046 +0000 UTC m=+603.166367916" watchObservedRunningTime="2026-07-09 15:41:21.44613982 +0000 UTC m=+603.167646666" Jul 09 15:41:21.465256 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:41:21.465191 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-6c45ffb8d8-6x46q" podStartSLOduration=0.972806483 podStartE2EDuration="3.465142388s" totalImagesPullingTime="2.492335905s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 15:41:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 15:41:21.464344674 +0000 UTC m=+603.185851545" watchObservedRunningTime="2026-07-09 15:41:21.465142388 +0000 UTC m=+603.186649258" Jul 09 15:41:21.484167 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:41:21.484117 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-58f86c7798-jkj5d" podStartSLOduration=1.59078884 podStartE2EDuration="3.484098141s" totalImagesPullingTime="1.893309301s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 15:41:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 15:41:21.481973376 +0000 UTC m=+603.203480246" watchObservedRunningTime="2026-07-09 15:41:21.484098141 +0000 UTC m=+603.205605010" Jul 09 15:41:27.438149 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:41:27.438120 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-6c45ffb8d8-6x46q" Jul 09 15:41:32.435063 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:41:32.435028 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-58f86c7798-jkj5d" Jul 09 15:41:32.435512 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:41:32.435277 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-7964457696-m5gbg" Jul 09 15:41:38.111124 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:41:38.111092 2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-bkcwb"] Jul 09 15:41:38.115202 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:41:38.115177 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-bkcwb" Jul 09 15:41:38.118682 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:41:38.118659 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\"" Jul 09 15:41:38.118821 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:41:38.118803 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-4b87c\"" Jul 09 15:41:38.119252 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:41:38.119239 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\"" Jul 09 15:41:38.119939 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:41:38.119923 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-distinct-2-simulated-kserve-self-signed-certs\"" Jul 09 15:41:38.130147 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:41:38.130122 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-bkcwb"] Jul 09 15:41:38.205614 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:41:38.205575 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mqgkw\" (UniqueName: \"kubernetes.io/projected/69646e64-e81d-4dd0-99c4-408a27cc16b7-kube-api-access-mqgkw\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-bkcwb\" (UID: \"69646e64-e81d-4dd0-99c4-408a27cc16b7\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-bkcwb" Jul 09 15:41:38.205614 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:41:38.205621 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/69646e64-e81d-4dd0-99c4-408a27cc16b7-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-bkcwb\" (UID: \"69646e64-e81d-4dd0-99c4-408a27cc16b7\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-bkcwb" Jul 09 15:41:38.205824 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:41:38.205649 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/69646e64-e81d-4dd0-99c4-408a27cc16b7-model-cache\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-bkcwb\" (UID: \"69646e64-e81d-4dd0-99c4-408a27cc16b7\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-bkcwb" Jul 09 15:41:38.205824 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:41:38.205690 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/69646e64-e81d-4dd0-99c4-408a27cc16b7-home\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-bkcwb\" (UID: \"69646e64-e81d-4dd0-99c4-408a27cc16b7\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-bkcwb" Jul 09 15:41:38.205824 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:41:38.205731 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/69646e64-e81d-4dd0-99c4-408a27cc16b7-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-bkcwb\" (UID: \"69646e64-e81d-4dd0-99c4-408a27cc16b7\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-bkcwb" Jul 09 15:41:38.205824 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:41:38.205753 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/69646e64-e81d-4dd0-99c4-408a27cc16b7-dshm\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-bkcwb\" (UID: \"69646e64-e81d-4dd0-99c4-408a27cc16b7\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-bkcwb" Jul 09 15:41:38.306860 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:41:38.306817 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mqgkw\" (UniqueName: \"kubernetes.io/projected/69646e64-e81d-4dd0-99c4-408a27cc16b7-kube-api-access-mqgkw\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-bkcwb\" (UID: \"69646e64-e81d-4dd0-99c4-408a27cc16b7\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-bkcwb" Jul 09 15:41:38.306860 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:41:38.306863 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/69646e64-e81d-4dd0-99c4-408a27cc16b7-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-bkcwb\" (UID: \"69646e64-e81d-4dd0-99c4-408a27cc16b7\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-bkcwb" Jul 09 15:41:38.307119 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:41:38.306909 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/69646e64-e81d-4dd0-99c4-408a27cc16b7-model-cache\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-bkcwb\" (UID: \"69646e64-e81d-4dd0-99c4-408a27cc16b7\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-bkcwb" Jul 09 15:41:38.307119 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:41:38.306952 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/69646e64-e81d-4dd0-99c4-408a27cc16b7-home\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-bkcwb\" (UID: \"69646e64-e81d-4dd0-99c4-408a27cc16b7\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-bkcwb" Jul 09 15:41:38.307119 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:41:38.307006 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/69646e64-e81d-4dd0-99c4-408a27cc16b7-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-bkcwb\" (UID: \"69646e64-e81d-4dd0-99c4-408a27cc16b7\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-bkcwb" Jul 09 15:41:38.307119 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:41:38.307044 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/69646e64-e81d-4dd0-99c4-408a27cc16b7-dshm\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-bkcwb\" (UID: \"69646e64-e81d-4dd0-99c4-408a27cc16b7\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-bkcwb" Jul 09 15:41:38.307388 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:41:38.307366 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/69646e64-e81d-4dd0-99c4-408a27cc16b7-model-cache\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-bkcwb\" (UID: \"69646e64-e81d-4dd0-99c4-408a27cc16b7\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-bkcwb" Jul 09 15:41:38.307456 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:41:38.307425 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/69646e64-e81d-4dd0-99c4-408a27cc16b7-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-bkcwb\" (UID: \"69646e64-e81d-4dd0-99c4-408a27cc16b7\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-bkcwb" Jul 09 15:41:38.307517 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:41:38.307438 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/69646e64-e81d-4dd0-99c4-408a27cc16b7-home\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-bkcwb\" (UID: \"69646e64-e81d-4dd0-99c4-408a27cc16b7\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-bkcwb" Jul 09 15:41:38.309328 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:41:38.309307 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/69646e64-e81d-4dd0-99c4-408a27cc16b7-dshm\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-bkcwb\" (UID: \"69646e64-e81d-4dd0-99c4-408a27cc16b7\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-bkcwb" Jul 09 15:41:38.309478 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:41:38.309461 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/69646e64-e81d-4dd0-99c4-408a27cc16b7-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-bkcwb\" (UID: \"69646e64-e81d-4dd0-99c4-408a27cc16b7\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-bkcwb" Jul 09 15:41:38.317622 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:41:38.317594 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mqgkw\" (UniqueName: \"kubernetes.io/projected/69646e64-e81d-4dd0-99c4-408a27cc16b7-kube-api-access-mqgkw\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-bkcwb\" (UID: \"69646e64-e81d-4dd0-99c4-408a27cc16b7\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-bkcwb" Jul 09 15:41:38.425146 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:41:38.425052 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-bkcwb" Jul 09 15:41:38.566234 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:41:38.566210 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-bkcwb"] Jul 09 15:41:38.568774 ip-10-0-133-183 kubenswrapper[2573]: W0709 15:41:38.568744 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod69646e64_e81d_4dd0_99c4_408a27cc16b7.slice/crio-a52a78f51691a06f77c245a614f7324cd25ead5327de70467b6bdd7923b91c38 WatchSource:0}: Error finding container a52a78f51691a06f77c245a614f7324cd25ead5327de70467b6bdd7923b91c38: Status 404 returned error can't find the container with id a52a78f51691a06f77c245a614f7324cd25ead5327de70467b6bdd7923b91c38 Jul 09 15:41:38.570219 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:41:38.570201 2573 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 09 15:41:39.514761 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:41:39.514724 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-bkcwb" event={"ID":"69646e64-e81d-4dd0-99c4-408a27cc16b7","Type":"ContainerStarted","Data":"a52a78f51691a06f77c245a614f7324cd25ead5327de70467b6bdd7923b91c38"} Jul 09 15:41:40.520302 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:41:40.520265 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-bkcwb" event={"ID":"69646e64-e81d-4dd0-99c4-408a27cc16b7","Type":"ContainerStarted","Data":"526bbdaf9f8ea3009b5c6175ff9de48d7250bb0cd5bd5077cb19f6bb56bc7c92"} Jul 09 15:41:40.544679 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:41:40.544630 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-bkcwb" podStartSLOduration=1.350138683 podStartE2EDuration="2.544610108s" totalImagesPullingTime="1.194471425s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 15:41:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 15:41:40.542391448 +0000 UTC m=+622.263898320" watchObservedRunningTime="2026-07-09 15:41:40.544610108 +0000 UTC m=+622.266116977" Jul 09 15:41:43.811599 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:41:43.811551 2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-distinct-simulated-kserve-59975d69f9-zfnbz"] Jul 09 15:41:43.815188 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:41:43.815165 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-zfnbz" Jul 09 15:41:43.818187 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:41:43.818163 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-distinct-simulated-kserve-self-signed-certs\"" Jul 09 15:41:43.832025 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:41:43.831996 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-simulated-kserve-59975d69f9-zfnbz"] Jul 09 15:41:43.960290 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:41:43.960247 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/feacfd9b-35b5-40bf-88cf-8ff322bde482-home\") pod \"e2e-distinct-simulated-kserve-59975d69f9-zfnbz\" (UID: \"feacfd9b-35b5-40bf-88cf-8ff322bde482\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-zfnbz" Jul 09 15:41:43.960290 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:41:43.960290 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/feacfd9b-35b5-40bf-88cf-8ff322bde482-tls-certs\") pod \"e2e-distinct-simulated-kserve-59975d69f9-zfnbz\" (UID: \"feacfd9b-35b5-40bf-88cf-8ff322bde482\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-zfnbz" Jul 09 15:41:43.960505 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:41:43.960322 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wffj4\" (UniqueName: \"kubernetes.io/projected/feacfd9b-35b5-40bf-88cf-8ff322bde482-kube-api-access-wffj4\") pod \"e2e-distinct-simulated-kserve-59975d69f9-zfnbz\" (UID: \"feacfd9b-35b5-40bf-88cf-8ff322bde482\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-zfnbz" Jul 09 15:41:43.960505 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:41:43.960422 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/feacfd9b-35b5-40bf-88cf-8ff322bde482-dshm\") pod \"e2e-distinct-simulated-kserve-59975d69f9-zfnbz\" (UID: \"feacfd9b-35b5-40bf-88cf-8ff322bde482\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-zfnbz" Jul 09 15:41:43.960505 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:41:43.960449 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/feacfd9b-35b5-40bf-88cf-8ff322bde482-model-cache\") pod \"e2e-distinct-simulated-kserve-59975d69f9-zfnbz\" (UID: \"feacfd9b-35b5-40bf-88cf-8ff322bde482\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-zfnbz" Jul 09 15:41:43.960621 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:41:43.960522 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/feacfd9b-35b5-40bf-88cf-8ff322bde482-tmp-dir\") pod \"e2e-distinct-simulated-kserve-59975d69f9-zfnbz\" (UID: \"feacfd9b-35b5-40bf-88cf-8ff322bde482\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-zfnbz" Jul 09 15:41:44.061188 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:41:44.061145 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/feacfd9b-35b5-40bf-88cf-8ff322bde482-dshm\") pod \"e2e-distinct-simulated-kserve-59975d69f9-zfnbz\" (UID: \"feacfd9b-35b5-40bf-88cf-8ff322bde482\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-zfnbz" Jul 09 15:41:44.061188 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:41:44.061186 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/feacfd9b-35b5-40bf-88cf-8ff322bde482-model-cache\") pod \"e2e-distinct-simulated-kserve-59975d69f9-zfnbz\" (UID: \"feacfd9b-35b5-40bf-88cf-8ff322bde482\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-zfnbz" Jul 09 15:41:44.061449 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:41:44.061229 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/feacfd9b-35b5-40bf-88cf-8ff322bde482-tmp-dir\") pod \"e2e-distinct-simulated-kserve-59975d69f9-zfnbz\" (UID: \"feacfd9b-35b5-40bf-88cf-8ff322bde482\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-zfnbz" Jul 09 15:41:44.061449 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:41:44.061285 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/feacfd9b-35b5-40bf-88cf-8ff322bde482-home\") pod \"e2e-distinct-simulated-kserve-59975d69f9-zfnbz\" (UID: \"feacfd9b-35b5-40bf-88cf-8ff322bde482\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-zfnbz" Jul 09 15:41:44.061449 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:41:44.061306 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/feacfd9b-35b5-40bf-88cf-8ff322bde482-tls-certs\") pod \"e2e-distinct-simulated-kserve-59975d69f9-zfnbz\" (UID: \"feacfd9b-35b5-40bf-88cf-8ff322bde482\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-zfnbz" Jul 09 15:41:44.061449 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:41:44.061338 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wffj4\" (UniqueName: \"kubernetes.io/projected/feacfd9b-35b5-40bf-88cf-8ff322bde482-kube-api-access-wffj4\") pod \"e2e-distinct-simulated-kserve-59975d69f9-zfnbz\" (UID: \"feacfd9b-35b5-40bf-88cf-8ff322bde482\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-zfnbz" Jul 09 15:41:44.061730 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:41:44.061671 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/feacfd9b-35b5-40bf-88cf-8ff322bde482-model-cache\") pod \"e2e-distinct-simulated-kserve-59975d69f9-zfnbz\" (UID: \"feacfd9b-35b5-40bf-88cf-8ff322bde482\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-zfnbz" Jul 09 15:41:44.061730 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:41:44.061696 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/feacfd9b-35b5-40bf-88cf-8ff322bde482-tmp-dir\") pod \"e2e-distinct-simulated-kserve-59975d69f9-zfnbz\" (UID: \"feacfd9b-35b5-40bf-88cf-8ff322bde482\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-zfnbz" Jul 09 15:41:44.061827 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:41:44.061736 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/feacfd9b-35b5-40bf-88cf-8ff322bde482-home\") pod \"e2e-distinct-simulated-kserve-59975d69f9-zfnbz\" (UID: \"feacfd9b-35b5-40bf-88cf-8ff322bde482\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-zfnbz" Jul 09 15:41:44.063456 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:41:44.063439 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/feacfd9b-35b5-40bf-88cf-8ff322bde482-dshm\") pod \"e2e-distinct-simulated-kserve-59975d69f9-zfnbz\" (UID: \"feacfd9b-35b5-40bf-88cf-8ff322bde482\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-zfnbz" Jul 09 15:41:44.063766 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:41:44.063749 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/feacfd9b-35b5-40bf-88cf-8ff322bde482-tls-certs\") pod \"e2e-distinct-simulated-kserve-59975d69f9-zfnbz\" (UID: \"feacfd9b-35b5-40bf-88cf-8ff322bde482\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-zfnbz" Jul 09 15:41:44.072025 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:41:44.072002 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wffj4\" (UniqueName: \"kubernetes.io/projected/feacfd9b-35b5-40bf-88cf-8ff322bde482-kube-api-access-wffj4\") pod \"e2e-distinct-simulated-kserve-59975d69f9-zfnbz\" (UID: \"feacfd9b-35b5-40bf-88cf-8ff322bde482\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-zfnbz" Jul 09 15:41:44.125389 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:41:44.125351 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-zfnbz" Jul 09 15:41:44.266340 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:41:44.266311 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-simulated-kserve-59975d69f9-zfnbz"] Jul 09 15:41:44.269027 ip-10-0-133-183 kubenswrapper[2573]: W0709 15:41:44.269000 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfeacfd9b_35b5_40bf_88cf_8ff322bde482.slice/crio-c415e469fbe5e3b3772eab8545442d75665e1410801f640feb1d160b688b083b WatchSource:0}: Error finding container c415e469fbe5e3b3772eab8545442d75665e1410801f640feb1d160b688b083b: Status 404 returned error can't find the container with id c415e469fbe5e3b3772eab8545442d75665e1410801f640feb1d160b688b083b Jul 09 15:41:44.540656 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:41:44.540629 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-zfnbz" event={"ID":"feacfd9b-35b5-40bf-88cf-8ff322bde482","Type":"ContainerStarted","Data":"c415e469fbe5e3b3772eab8545442d75665e1410801f640feb1d160b688b083b"} Jul 09 15:41:45.546345 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:41:45.546301 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-zfnbz" event={"ID":"feacfd9b-35b5-40bf-88cf-8ff322bde482","Type":"ContainerStarted","Data":"8aa430b208aa04d61d6dbb2e3f8d3e3e3bdca3cebeebe4cc74518f72956f1158"} Jul 09 15:41:45.591535 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:41:45.591481 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-zfnbz" podStartSLOduration=2.363220216 podStartE2EDuration="2.59146264s" totalImagesPullingTime="228.242424ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 15:41:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 15:41:45.589948864 +0000 UTC m=+627.311455733" watchObservedRunningTime="2026-07-09 15:41:45.59146264 +0000 UTC m=+627.312969543" Jul 09 15:41:48.425989 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:41:48.425951 2573 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-bkcwb" Jul 09 15:41:48.426404 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:41:48.426035 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-bkcwb" Jul 09 15:41:48.438706 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:41:48.438680 2573 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-bkcwb" Jul 09 15:41:48.573029 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:41:48.573005 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-bkcwb" Jul 09 15:41:54.126395 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:41:54.126358 2573 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-zfnbz" Jul 09 15:41:54.126781 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:41:54.126449 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-zfnbz" Jul 09 15:41:54.138664 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:41:54.138641 2573 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-zfnbz" Jul 09 15:41:54.601401 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:41:54.601374 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-zfnbz" Jul 09 15:45:12.782483 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:45:12.782389 2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-647758b7f9-fgkbc"] Jul 09 15:45:12.783088 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:45:12.782626 2573 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/maas-controller-647758b7f9-fgkbc" podUID="47bb4e59-faa2-443d-85ce-711d6a4a157e" containerName="manager" containerID="cri-o://cd9094c792de52ac6225be10dbaf6e0ef55992f462a4c89141462fdeccc5641f" gracePeriod=30 Jul 09 15:45:13.030385 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:45:13.030357 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-647758b7f9-fgkbc" Jul 09 15:45:13.121808 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:45:13.121710 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5n92k\" (UniqueName: \"kubernetes.io/projected/47bb4e59-faa2-443d-85ce-711d6a4a157e-kube-api-access-5n92k\") pod \"47bb4e59-faa2-443d-85ce-711d6a4a157e\" (UID: \"47bb4e59-faa2-443d-85ce-711d6a4a157e\") " Jul 09 15:45:13.121996 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:45:13.121919 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/47bb4e59-faa2-443d-85ce-711d6a4a157e-webhook-cert\") pod \"47bb4e59-faa2-443d-85ce-711d6a4a157e\" (UID: \"47bb4e59-faa2-443d-85ce-711d6a4a157e\") " Jul 09 15:45:13.123767 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:45:13.123726 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/47bb4e59-faa2-443d-85ce-711d6a4a157e-kube-api-access-5n92k" (OuterVolumeSpecName: "kube-api-access-5n92k") pod "47bb4e59-faa2-443d-85ce-711d6a4a157e" (UID: "47bb4e59-faa2-443d-85ce-711d6a4a157e"). InnerVolumeSpecName "kube-api-access-5n92k". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 09 15:45:13.123906 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:45:13.123813 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/47bb4e59-faa2-443d-85ce-711d6a4a157e-webhook-cert" (OuterVolumeSpecName: "webhook-cert") pod "47bb4e59-faa2-443d-85ce-711d6a4a157e" (UID: "47bb4e59-faa2-443d-85ce-711d6a4a157e"). InnerVolumeSpecName "webhook-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 09 15:45:13.222936 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:45:13.222870 2573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5n92k\" (UniqueName: \"kubernetes.io/projected/47bb4e59-faa2-443d-85ce-711d6a4a157e-kube-api-access-5n92k\") on node \"ip-10-0-133-183.ec2.internal\" DevicePath \"\"" Jul 09 15:45:13.222936 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:45:13.222931 2573 reconciler_common.go:299] "Volume detached for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/47bb4e59-faa2-443d-85ce-711d6a4a157e-webhook-cert\") on node \"ip-10-0-133-183.ec2.internal\" DevicePath \"\"" Jul 09 15:45:13.499473 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:45:13.499439 2573 generic.go:356] "Generic (PLEG): container finished" podID="47bb4e59-faa2-443d-85ce-711d6a4a157e" containerID="cd9094c792de52ac6225be10dbaf6e0ef55992f462a4c89141462fdeccc5641f" exitCode=0 Jul 09 15:45:13.499690 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:45:13.499495 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-647758b7f9-fgkbc" Jul 09 15:45:13.499690 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:45:13.499530 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-647758b7f9-fgkbc" event={"ID":"47bb4e59-faa2-443d-85ce-711d6a4a157e","Type":"ContainerDied","Data":"cd9094c792de52ac6225be10dbaf6e0ef55992f462a4c89141462fdeccc5641f"} Jul 09 15:45:13.499690 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:45:13.499596 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-647758b7f9-fgkbc" event={"ID":"47bb4e59-faa2-443d-85ce-711d6a4a157e","Type":"ContainerDied","Data":"fc6daff865690746506dba360afe23cea9d5f052150f82ed275129cf599d8e86"} Jul 09 15:45:13.499690 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:45:13.499621 2573 scope.go:117] "RemoveContainer" containerID="cd9094c792de52ac6225be10dbaf6e0ef55992f462a4c89141462fdeccc5641f" Jul 09 15:45:13.508551 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:45:13.508536 2573 scope.go:117] "RemoveContainer" containerID="cd9094c792de52ac6225be10dbaf6e0ef55992f462a4c89141462fdeccc5641f" Jul 09 15:45:13.508822 ip-10-0-133-183 kubenswrapper[2573]: E0709 15:45:13.508802 2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cd9094c792de52ac6225be10dbaf6e0ef55992f462a4c89141462fdeccc5641f\": container with ID starting with cd9094c792de52ac6225be10dbaf6e0ef55992f462a4c89141462fdeccc5641f not found: ID does not exist" containerID="cd9094c792de52ac6225be10dbaf6e0ef55992f462a4c89141462fdeccc5641f" Jul 09 15:45:13.508916 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:45:13.508833 2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cd9094c792de52ac6225be10dbaf6e0ef55992f462a4c89141462fdeccc5641f"} err="failed to get container status \"cd9094c792de52ac6225be10dbaf6e0ef55992f462a4c89141462fdeccc5641f\": rpc error: code = NotFound desc = could not find container \"cd9094c792de52ac6225be10dbaf6e0ef55992f462a4c89141462fdeccc5641f\": container with ID starting with cd9094c792de52ac6225be10dbaf6e0ef55992f462a4c89141462fdeccc5641f not found: ID does not exist" Jul 09 15:45:13.524208 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:45:13.524184 2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-647758b7f9-fgkbc"] Jul 09 15:45:13.527364 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:45:13.527334 2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-controller-647758b7f9-fgkbc"] Jul 09 15:45:14.376864 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:45:14.376831 2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-647758b7f9-bgl5w"] Jul 09 15:45:14.377254 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:45:14.377222 2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="47bb4e59-faa2-443d-85ce-711d6a4a157e" containerName="manager" Jul 09 15:45:14.377254 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:45:14.377239 2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="47bb4e59-faa2-443d-85ce-711d6a4a157e" containerName="manager" Jul 09 15:45:14.377440 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:45:14.377345 2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="47bb4e59-faa2-443d-85ce-711d6a4a157e" containerName="manager" Jul 09 15:45:14.381573 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:45:14.381556 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-647758b7f9-bgl5w" Jul 09 15:45:14.386402 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:45:14.386383 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-vgq75\"" Jul 09 15:45:14.386402 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:45:14.386383 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\"" Jul 09 15:45:14.386566 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:45:14.386385 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\"" Jul 09 15:45:14.391836 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:45:14.391810 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-647758b7f9-bgl5w"] Jul 09 15:45:14.433186 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:45:14.433157 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kf2qs\" (UniqueName: \"kubernetes.io/projected/6af05785-c895-415e-923d-23ee28c4eece-kube-api-access-kf2qs\") pod \"maas-controller-647758b7f9-bgl5w\" (UID: \"6af05785-c895-415e-923d-23ee28c4eece\") " pod="opendatahub/maas-controller-647758b7f9-bgl5w" Jul 09 15:45:14.433362 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:45:14.433204 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/6af05785-c895-415e-923d-23ee28c4eece-webhook-cert\") pod \"maas-controller-647758b7f9-bgl5w\" (UID: \"6af05785-c895-415e-923d-23ee28c4eece\") " pod="opendatahub/maas-controller-647758b7f9-bgl5w" Jul 09 15:45:14.534133 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:45:14.534099 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kf2qs\" (UniqueName: \"kubernetes.io/projected/6af05785-c895-415e-923d-23ee28c4eece-kube-api-access-kf2qs\") pod \"maas-controller-647758b7f9-bgl5w\" (UID: \"6af05785-c895-415e-923d-23ee28c4eece\") " pod="opendatahub/maas-controller-647758b7f9-bgl5w" Jul 09 15:45:14.534313 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:45:14.534170 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/6af05785-c895-415e-923d-23ee28c4eece-webhook-cert\") pod \"maas-controller-647758b7f9-bgl5w\" (UID: \"6af05785-c895-415e-923d-23ee28c4eece\") " pod="opendatahub/maas-controller-647758b7f9-bgl5w" Jul 09 15:45:14.536638 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:45:14.536618 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/6af05785-c895-415e-923d-23ee28c4eece-webhook-cert\") pod \"maas-controller-647758b7f9-bgl5w\" (UID: \"6af05785-c895-415e-923d-23ee28c4eece\") " pod="opendatahub/maas-controller-647758b7f9-bgl5w" Jul 09 15:45:14.544502 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:45:14.544475 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kf2qs\" (UniqueName: \"kubernetes.io/projected/6af05785-c895-415e-923d-23ee28c4eece-kube-api-access-kf2qs\") pod \"maas-controller-647758b7f9-bgl5w\" (UID: \"6af05785-c895-415e-923d-23ee28c4eece\") " pod="opendatahub/maas-controller-647758b7f9-bgl5w" Jul 09 15:45:14.693200 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:45:14.693157 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-647758b7f9-bgl5w" Jul 09 15:45:14.812000 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:45:14.811964 2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="47bb4e59-faa2-443d-85ce-711d6a4a157e" path="/var/lib/kubelet/pods/47bb4e59-faa2-443d-85ce-711d6a4a157e/volumes" Jul 09 15:45:14.834864 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:45:14.834842 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-647758b7f9-bgl5w"] Jul 09 15:45:14.837078 ip-10-0-133-183 kubenswrapper[2573]: W0709 15:45:14.837053 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6af05785_c895_415e_923d_23ee28c4eece.slice/crio-40d2cd7323480b599a345ea5ae97b81fe552bb882e6741fdd1c09646d9530d68 WatchSource:0}: Error finding container 40d2cd7323480b599a345ea5ae97b81fe552bb882e6741fdd1c09646d9530d68: Status 404 returned error can't find the container with id 40d2cd7323480b599a345ea5ae97b81fe552bb882e6741fdd1c09646d9530d68 Jul 09 15:45:15.517165 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:45:15.517124 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-647758b7f9-bgl5w" event={"ID":"6af05785-c895-415e-923d-23ee28c4eece","Type":"ContainerStarted","Data":"40d2cd7323480b599a345ea5ae97b81fe552bb882e6741fdd1c09646d9530d68"} Jul 09 15:45:16.523834 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:45:16.523799 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-647758b7f9-bgl5w" event={"ID":"6af05785-c895-415e-923d-23ee28c4eece","Type":"ContainerStarted","Data":"0d98df55cee4daf8cafdda5c6af483e5b4cfa07ab3efc3ef65b9554510216166"} Jul 09 15:45:16.524227 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:45:16.523888 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-647758b7f9-bgl5w" Jul 09 15:45:16.548616 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:45:16.548555 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-controller-647758b7f9-bgl5w" podStartSLOduration=1.642244193 podStartE2EDuration="2.548535957s" totalImagesPullingTime="906.291764ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 15:45:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 15:45:16.547328311 +0000 UTC m=+838.268835181" watchObservedRunningTime="2026-07-09 15:45:16.548535957 +0000 UTC m=+838.270042826" Jul 09 15:45:27.535229 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:45:27.535197 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-647758b7f9-bgl5w" Jul 09 15:46:18.713294 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:46:18.713209 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-b7b9v_db22f88d-a953-488e-b384-67f991351f73/ovn-acl-logging/0.log" Jul 09 15:46:18.722419 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:46:18.722393 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-b7b9v_db22f88d-a953-488e-b384-67f991351f73/ovn-acl-logging/0.log" Jul 09 15:51:18.742707 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:51:18.742634 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-b7b9v_db22f88d-a953-488e-b384-67f991351f73/ovn-acl-logging/0.log" Jul 09 15:51:18.752047 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:51:18.752021 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-b7b9v_db22f88d-a953-488e-b384-67f991351f73/ovn-acl-logging/0.log" Jul 09 15:56:18.771336 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:56:18.771307 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-b7b9v_db22f88d-a953-488e-b384-67f991351f73/ovn-acl-logging/0.log" Jul 09 15:56:18.780987 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:56:18.780966 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-b7b9v_db22f88d-a953-488e-b384-67f991351f73/ovn-acl-logging/0.log" Jul 09 15:57:27.246969 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:57:27.246930 2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-077a59c2-5fbbc94679-l7vq2"] Jul 09 15:57:27.250919 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:57:27.250897 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-077a59c2-5fbbc94679-l7vq2" Jul 09 15:57:27.253464 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:57:27.253445 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ait-077a59c2-serving-cert\"" Jul 09 15:57:27.259398 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:57:27.259369 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-077a59c2-5fbbc94679-l7vq2"] Jul 09 15:57:27.339276 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:57:27.339244 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gsr4r\" (UniqueName: \"kubernetes.io/projected/75346131-f4ad-4a1b-876f-f3acd21d47f3-kube-api-access-gsr4r\") pod \"maas-api-e2e-ait-077a59c2-5fbbc94679-l7vq2\" (UID: \"75346131-f4ad-4a1b-876f-f3acd21d47f3\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-077a59c2-5fbbc94679-l7vq2" Jul 09 15:57:27.339467 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:57:27.339296 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/75346131-f4ad-4a1b-876f-f3acd21d47f3-openshift-service-ca\") pod \"maas-api-e2e-ait-077a59c2-5fbbc94679-l7vq2\" (UID: \"75346131-f4ad-4a1b-876f-f3acd21d47f3\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-077a59c2-5fbbc94679-l7vq2" Jul 09 15:57:27.339467 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:57:27.339334 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/75346131-f4ad-4a1b-876f-f3acd21d47f3-maas-api-tls\") pod \"maas-api-e2e-ait-077a59c2-5fbbc94679-l7vq2\" (UID: \"75346131-f4ad-4a1b-876f-f3acd21d47f3\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-077a59c2-5fbbc94679-l7vq2" Jul 09 15:57:27.440380 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:57:27.440347 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/75346131-f4ad-4a1b-876f-f3acd21d47f3-openshift-service-ca\") pod \"maas-api-e2e-ait-077a59c2-5fbbc94679-l7vq2\" (UID: \"75346131-f4ad-4a1b-876f-f3acd21d47f3\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-077a59c2-5fbbc94679-l7vq2" Jul 09 15:57:27.440555 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:57:27.440400 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/75346131-f4ad-4a1b-876f-f3acd21d47f3-maas-api-tls\") pod \"maas-api-e2e-ait-077a59c2-5fbbc94679-l7vq2\" (UID: \"75346131-f4ad-4a1b-876f-f3acd21d47f3\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-077a59c2-5fbbc94679-l7vq2" Jul 09 15:57:27.440555 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:57:27.440449 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gsr4r\" (UniqueName: \"kubernetes.io/projected/75346131-f4ad-4a1b-876f-f3acd21d47f3-kube-api-access-gsr4r\") pod \"maas-api-e2e-ait-077a59c2-5fbbc94679-l7vq2\" (UID: \"75346131-f4ad-4a1b-876f-f3acd21d47f3\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-077a59c2-5fbbc94679-l7vq2" Jul 09 15:57:27.441141 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:57:27.441120 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/75346131-f4ad-4a1b-876f-f3acd21d47f3-openshift-service-ca\") pod \"maas-api-e2e-ait-077a59c2-5fbbc94679-l7vq2\" (UID: \"75346131-f4ad-4a1b-876f-f3acd21d47f3\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-077a59c2-5fbbc94679-l7vq2" Jul 09 15:57:27.442917 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:57:27.442895 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/75346131-f4ad-4a1b-876f-f3acd21d47f3-maas-api-tls\") pod \"maas-api-e2e-ait-077a59c2-5fbbc94679-l7vq2\" (UID: \"75346131-f4ad-4a1b-876f-f3acd21d47f3\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-077a59c2-5fbbc94679-l7vq2" Jul 09 15:57:27.449608 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:57:27.449576 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gsr4r\" (UniqueName: \"kubernetes.io/projected/75346131-f4ad-4a1b-876f-f3acd21d47f3-kube-api-access-gsr4r\") pod \"maas-api-e2e-ait-077a59c2-5fbbc94679-l7vq2\" (UID: \"75346131-f4ad-4a1b-876f-f3acd21d47f3\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-077a59c2-5fbbc94679-l7vq2" Jul 09 15:57:27.562668 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:57:27.562584 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-077a59c2-5fbbc94679-l7vq2" Jul 09 15:57:27.712891 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:57:27.712666 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-077a59c2-5fbbc94679-l7vq2"] Jul 09 15:57:27.715989 ip-10-0-133-183 kubenswrapper[2573]: W0709 15:57:27.714095 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod75346131_f4ad_4a1b_876f_f3acd21d47f3.slice/crio-b84ddc1aab5db09feb8c0d37c32826c75f6be9ec6ed7b6e1d6568753b8ba0764 WatchSource:0}: Error finding container b84ddc1aab5db09feb8c0d37c32826c75f6be9ec6ed7b6e1d6568753b8ba0764: Status 404 returned error can't find the container with id b84ddc1aab5db09feb8c0d37c32826c75f6be9ec6ed7b6e1d6568753b8ba0764 Jul 09 15:57:27.717739 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:57:27.717719 2573 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 09 15:57:27.895408 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:57:27.895320 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-077a59c2-5fbbc94679-l7vq2" event={"ID":"75346131-f4ad-4a1b-876f-f3acd21d47f3","Type":"ContainerStarted","Data":"b84ddc1aab5db09feb8c0d37c32826c75f6be9ec6ed7b6e1d6568753b8ba0764"} Jul 09 15:57:29.906435 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:57:29.906339 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-077a59c2-5fbbc94679-l7vq2" event={"ID":"75346131-f4ad-4a1b-876f-f3acd21d47f3","Type":"ContainerStarted","Data":"d59e4f07227f1854d7ad727eedf35831e39f7431841c2a53265d2ee14903c6e2"} Jul 09 15:57:29.906435 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:57:29.906396 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-077a59c2-5fbbc94679-l7vq2" Jul 09 15:57:29.925736 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:57:29.925681 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ait-077a59c2-5fbbc94679-l7vq2" podStartSLOduration=1.06095708 podStartE2EDuration="2.925666577s" totalImagesPullingTime="1.864709497s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 15:57:27 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 15:57:29.92348745 +0000 UTC m=+1571.644994341" watchObservedRunningTime="2026-07-09 15:57:29.925666577 +0000 UTC m=+1571.647173444" Jul 09 15:57:32.784377 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:57:32.784345 2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-077a59c2-5fbbc94679-l7vq2"] Jul 09 15:57:32.784751 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:57:32.784542 2573 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ait-077a59c2-5fbbc94679-l7vq2" podUID="75346131-f4ad-4a1b-876f-f3acd21d47f3" containerName="maas-api" containerID="cri-o://d59e4f07227f1854d7ad727eedf35831e39f7431841c2a53265d2ee14903c6e2" gracePeriod=30 Jul 09 15:57:32.921810 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:57:32.921781 2573 generic.go:356] "Generic (PLEG): container finished" podID="75346131-f4ad-4a1b-876f-f3acd21d47f3" containerID="d59e4f07227f1854d7ad727eedf35831e39f7431841c2a53265d2ee14903c6e2" exitCode=0 Jul 09 15:57:32.921995 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:57:32.921832 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-077a59c2-5fbbc94679-l7vq2" event={"ID":"75346131-f4ad-4a1b-876f-f3acd21d47f3","Type":"ContainerDied","Data":"d59e4f07227f1854d7ad727eedf35831e39f7431841c2a53265d2ee14903c6e2"} Jul 09 15:57:33.033306 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:57:33.033278 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-077a59c2-5fbbc94679-l7vq2" Jul 09 15:57:33.197262 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:57:33.197228 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/75346131-f4ad-4a1b-876f-f3acd21d47f3-maas-api-tls\") pod \"75346131-f4ad-4a1b-876f-f3acd21d47f3\" (UID: \"75346131-f4ad-4a1b-876f-f3acd21d47f3\") " Jul 09 15:57:33.197395 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:57:33.197306 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-gsr4r\" (UniqueName: \"kubernetes.io/projected/75346131-f4ad-4a1b-876f-f3acd21d47f3-kube-api-access-gsr4r\") pod \"75346131-f4ad-4a1b-876f-f3acd21d47f3\" (UID: \"75346131-f4ad-4a1b-876f-f3acd21d47f3\") " Jul 09 15:57:33.197395 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:57:33.197334 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/75346131-f4ad-4a1b-876f-f3acd21d47f3-openshift-service-ca\") pod \"75346131-f4ad-4a1b-876f-f3acd21d47f3\" (UID: \"75346131-f4ad-4a1b-876f-f3acd21d47f3\") " Jul 09 15:57:33.197757 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:57:33.197720 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/75346131-f4ad-4a1b-876f-f3acd21d47f3-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "75346131-f4ad-4a1b-876f-f3acd21d47f3" (UID: "75346131-f4ad-4a1b-876f-f3acd21d47f3"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 09 15:57:33.199489 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:57:33.199462 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/75346131-f4ad-4a1b-876f-f3acd21d47f3-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "75346131-f4ad-4a1b-876f-f3acd21d47f3" (UID: "75346131-f4ad-4a1b-876f-f3acd21d47f3"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 09 15:57:33.199595 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:57:33.199489 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/75346131-f4ad-4a1b-876f-f3acd21d47f3-kube-api-access-gsr4r" (OuterVolumeSpecName: "kube-api-access-gsr4r") pod "75346131-f4ad-4a1b-876f-f3acd21d47f3" (UID: "75346131-f4ad-4a1b-876f-f3acd21d47f3"). InnerVolumeSpecName "kube-api-access-gsr4r". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 09 15:57:33.298620 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:57:33.298589 2573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gsr4r\" (UniqueName: \"kubernetes.io/projected/75346131-f4ad-4a1b-876f-f3acd21d47f3-kube-api-access-gsr4r\") on node \"ip-10-0-133-183.ec2.internal\" DevicePath \"\"" Jul 09 15:57:33.298620 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:57:33.298616 2573 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/75346131-f4ad-4a1b-876f-f3acd21d47f3-openshift-service-ca\") on node \"ip-10-0-133-183.ec2.internal\" DevicePath \"\"" Jul 09 15:57:33.298620 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:57:33.298627 2573 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/75346131-f4ad-4a1b-876f-f3acd21d47f3-maas-api-tls\") on node \"ip-10-0-133-183.ec2.internal\" DevicePath \"\"" Jul 09 15:57:33.927095 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:57:33.927065 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-077a59c2-5fbbc94679-l7vq2" Jul 09 15:57:33.927506 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:57:33.927115 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-077a59c2-5fbbc94679-l7vq2" event={"ID":"75346131-f4ad-4a1b-876f-f3acd21d47f3","Type":"ContainerDied","Data":"b84ddc1aab5db09feb8c0d37c32826c75f6be9ec6ed7b6e1d6568753b8ba0764"} Jul 09 15:57:33.927506 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:57:33.927177 2573 scope.go:117] "RemoveContainer" containerID="d59e4f07227f1854d7ad727eedf35831e39f7431841c2a53265d2ee14903c6e2" Jul 09 15:57:33.951270 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:57:33.951243 2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-077a59c2-5fbbc94679-l7vq2"] Jul 09 15:57:33.956041 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:57:33.956018 2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-077a59c2-5fbbc94679-l7vq2"] Jul 09 15:57:34.811442 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:57:34.811411 2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="75346131-f4ad-4a1b-876f-f3acd21d47f3" path="/var/lib/kubelet/pods/75346131-f4ad-4a1b-876f-f3acd21d47f3/volumes" Jul 09 15:57:40.356822 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:57:40.356773 2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-abf2097d-dddd5ff6-cq6ct"] Jul 09 15:57:40.357575 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:57:40.357546 2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="75346131-f4ad-4a1b-876f-f3acd21d47f3" containerName="maas-api" Jul 09 15:57:40.357575 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:57:40.357576 2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="75346131-f4ad-4a1b-876f-f3acd21d47f3" containerName="maas-api" Jul 09 15:57:40.357724 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:57:40.357707 2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="75346131-f4ad-4a1b-876f-f3acd21d47f3" containerName="maas-api" Jul 09 15:57:40.362440 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:57:40.362417 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-abf2097d-dddd5ff6-cq6ct" Jul 09 15:57:40.365111 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:57:40.365093 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ait-abf2097d-serving-cert\"" Jul 09 15:57:40.369474 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:57:40.369381 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-abf2097d-dddd5ff6-cq6ct"] Jul 09 15:57:40.461280 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:57:40.461246 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a1203443-2b22-4887-90be-5d4509a4626b-openshift-service-ca\") pod \"maas-api-e2e-ait-abf2097d-dddd5ff6-cq6ct\" (UID: \"a1203443-2b22-4887-90be-5d4509a4626b\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-abf2097d-dddd5ff6-cq6ct" Jul 09 15:57:40.461280 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:57:40.461283 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a1203443-2b22-4887-90be-5d4509a4626b-maas-api-tls\") pod \"maas-api-e2e-ait-abf2097d-dddd5ff6-cq6ct\" (UID: \"a1203443-2b22-4887-90be-5d4509a4626b\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-abf2097d-dddd5ff6-cq6ct" Jul 09 15:57:40.461490 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:57:40.461302 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fspx5\" (UniqueName: \"kubernetes.io/projected/a1203443-2b22-4887-90be-5d4509a4626b-kube-api-access-fspx5\") pod \"maas-api-e2e-ait-abf2097d-dddd5ff6-cq6ct\" (UID: \"a1203443-2b22-4887-90be-5d4509a4626b\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-abf2097d-dddd5ff6-cq6ct" Jul 09 15:57:40.562586 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:57:40.562548 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a1203443-2b22-4887-90be-5d4509a4626b-openshift-service-ca\") pod \"maas-api-e2e-ait-abf2097d-dddd5ff6-cq6ct\" (UID: \"a1203443-2b22-4887-90be-5d4509a4626b\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-abf2097d-dddd5ff6-cq6ct" Jul 09 15:57:40.562586 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:57:40.562589 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a1203443-2b22-4887-90be-5d4509a4626b-maas-api-tls\") pod \"maas-api-e2e-ait-abf2097d-dddd5ff6-cq6ct\" (UID: \"a1203443-2b22-4887-90be-5d4509a4626b\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-abf2097d-dddd5ff6-cq6ct" Jul 09 15:57:40.562830 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:57:40.562607 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fspx5\" (UniqueName: \"kubernetes.io/projected/a1203443-2b22-4887-90be-5d4509a4626b-kube-api-access-fspx5\") pod \"maas-api-e2e-ait-abf2097d-dddd5ff6-cq6ct\" (UID: \"a1203443-2b22-4887-90be-5d4509a4626b\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-abf2097d-dddd5ff6-cq6ct" Jul 09 15:57:40.562830 ip-10-0-133-183 kubenswrapper[2573]: E0709 15:57:40.562720 2573 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ait-abf2097d-serving-cert: secret "maas-api-e2e-ait-abf2097d-serving-cert" not found Jul 09 15:57:40.562830 ip-10-0-133-183 kubenswrapper[2573]: E0709 15:57:40.562785 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a1203443-2b22-4887-90be-5d4509a4626b-maas-api-tls podName:a1203443-2b22-4887-90be-5d4509a4626b nodeName:}" failed. No retries permitted until 2026-07-09 15:57:41.062766877 +0000 UTC m=+1582.784273728 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/a1203443-2b22-4887-90be-5d4509a4626b-maas-api-tls") pod "maas-api-e2e-ait-abf2097d-dddd5ff6-cq6ct" (UID: "a1203443-2b22-4887-90be-5d4509a4626b") : secret "maas-api-e2e-ait-abf2097d-serving-cert" not found Jul 09 15:57:40.563255 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:57:40.563235 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a1203443-2b22-4887-90be-5d4509a4626b-openshift-service-ca\") pod \"maas-api-e2e-ait-abf2097d-dddd5ff6-cq6ct\" (UID: \"a1203443-2b22-4887-90be-5d4509a4626b\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-abf2097d-dddd5ff6-cq6ct" Jul 09 15:57:40.572526 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:57:40.572499 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fspx5\" (UniqueName: \"kubernetes.io/projected/a1203443-2b22-4887-90be-5d4509a4626b-kube-api-access-fspx5\") pod \"maas-api-e2e-ait-abf2097d-dddd5ff6-cq6ct\" (UID: \"a1203443-2b22-4887-90be-5d4509a4626b\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-abf2097d-dddd5ff6-cq6ct" Jul 09 15:57:41.067205 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:57:41.067163 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a1203443-2b22-4887-90be-5d4509a4626b-maas-api-tls\") pod \"maas-api-e2e-ait-abf2097d-dddd5ff6-cq6ct\" (UID: \"a1203443-2b22-4887-90be-5d4509a4626b\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-abf2097d-dddd5ff6-cq6ct" Jul 09 15:57:41.069604 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:57:41.069582 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a1203443-2b22-4887-90be-5d4509a4626b-maas-api-tls\") pod \"maas-api-e2e-ait-abf2097d-dddd5ff6-cq6ct\" (UID: \"a1203443-2b22-4887-90be-5d4509a4626b\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-abf2097d-dddd5ff6-cq6ct" Jul 09 15:57:41.274445 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:57:41.274408 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-abf2097d-dddd5ff6-cq6ct" Jul 09 15:57:41.414112 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:57:41.414081 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-abf2097d-dddd5ff6-cq6ct"] Jul 09 15:57:41.416045 ip-10-0-133-183 kubenswrapper[2573]: W0709 15:57:41.416015 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda1203443_2b22_4887_90be_5d4509a4626b.slice/crio-4563876fe653918df855f57d3f3cae75e6843da914ec9775347d77e94fcc3223 WatchSource:0}: Error finding container 4563876fe653918df855f57d3f3cae75e6843da914ec9775347d77e94fcc3223: Status 404 returned error can't find the container with id 4563876fe653918df855f57d3f3cae75e6843da914ec9775347d77e94fcc3223 Jul 09 15:57:41.968659 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:57:41.968616 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-abf2097d-dddd5ff6-cq6ct" event={"ID":"a1203443-2b22-4887-90be-5d4509a4626b","Type":"ContainerStarted","Data":"aecafefe701156295c8b75044d857fdf50b5b989b1fccdfeb9ddd274120409fb"} Jul 09 15:57:41.968858 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:57:41.968684 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-abf2097d-dddd5ff6-cq6ct" event={"ID":"a1203443-2b22-4887-90be-5d4509a4626b","Type":"ContainerStarted","Data":"4563876fe653918df855f57d3f3cae75e6843da914ec9775347d77e94fcc3223"} Jul 09 15:57:41.968858 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:57:41.968850 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-abf2097d-dddd5ff6-cq6ct" Jul 09 15:57:41.988821 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:57:41.988760 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ait-abf2097d-dddd5ff6-cq6ct" podStartSLOduration=1.7323191310000001 podStartE2EDuration="1.988742184s" totalImagesPullingTime="256.423053ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 15:57:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 15:57:41.986948841 +0000 UTC m=+1583.708455710" watchObservedRunningTime="2026-07-09 15:57:41.988742184 +0000 UTC m=+1583.710249076" Jul 09 15:57:45.874945 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:57:45.874911 2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-abf2097d-dddd5ff6-cq6ct"] Jul 09 15:57:45.875396 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:57:45.875109 2573 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ait-abf2097d-dddd5ff6-cq6ct" podUID="a1203443-2b22-4887-90be-5d4509a4626b" containerName="maas-api" containerID="cri-o://aecafefe701156295c8b75044d857fdf50b5b989b1fccdfeb9ddd274120409fb" gracePeriod=30 Jul 09 15:57:46.121154 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:57:46.121129 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-abf2097d-dddd5ff6-cq6ct" Jul 09 15:57:46.215541 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:57:46.215504 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-fspx5\" (UniqueName: \"kubernetes.io/projected/a1203443-2b22-4887-90be-5d4509a4626b-kube-api-access-fspx5\") pod \"a1203443-2b22-4887-90be-5d4509a4626b\" (UID: \"a1203443-2b22-4887-90be-5d4509a4626b\") " Jul 09 15:57:46.215769 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:57:46.215568 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a1203443-2b22-4887-90be-5d4509a4626b-maas-api-tls\") pod \"a1203443-2b22-4887-90be-5d4509a4626b\" (UID: \"a1203443-2b22-4887-90be-5d4509a4626b\") " Jul 09 15:57:46.215769 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:57:46.215635 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a1203443-2b22-4887-90be-5d4509a4626b-openshift-service-ca\") pod \"a1203443-2b22-4887-90be-5d4509a4626b\" (UID: \"a1203443-2b22-4887-90be-5d4509a4626b\") " Jul 09 15:57:46.216070 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:57:46.216044 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a1203443-2b22-4887-90be-5d4509a4626b-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "a1203443-2b22-4887-90be-5d4509a4626b" (UID: "a1203443-2b22-4887-90be-5d4509a4626b"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 09 15:57:46.217659 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:57:46.217641 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a1203443-2b22-4887-90be-5d4509a4626b-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "a1203443-2b22-4887-90be-5d4509a4626b" (UID: "a1203443-2b22-4887-90be-5d4509a4626b"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 09 15:57:46.217725 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:57:46.217709 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a1203443-2b22-4887-90be-5d4509a4626b-kube-api-access-fspx5" (OuterVolumeSpecName: "kube-api-access-fspx5") pod "a1203443-2b22-4887-90be-5d4509a4626b" (UID: "a1203443-2b22-4887-90be-5d4509a4626b"). InnerVolumeSpecName "kube-api-access-fspx5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 09 15:57:46.317121 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:57:46.317068 2573 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a1203443-2b22-4887-90be-5d4509a4626b-openshift-service-ca\") on node \"ip-10-0-133-183.ec2.internal\" DevicePath \"\"" Jul 09 15:57:46.317121 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:57:46.317112 2573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fspx5\" (UniqueName: \"kubernetes.io/projected/a1203443-2b22-4887-90be-5d4509a4626b-kube-api-access-fspx5\") on node \"ip-10-0-133-183.ec2.internal\" DevicePath \"\"" Jul 09 15:57:46.317121 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:57:46.317123 2573 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a1203443-2b22-4887-90be-5d4509a4626b-maas-api-tls\") on node \"ip-10-0-133-183.ec2.internal\" DevicePath \"\"" Jul 09 15:57:46.994183 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:57:46.994152 2573 generic.go:356] "Generic (PLEG): container finished" podID="a1203443-2b22-4887-90be-5d4509a4626b" containerID="aecafefe701156295c8b75044d857fdf50b5b989b1fccdfeb9ddd274120409fb" exitCode=0 Jul 09 15:57:46.994577 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:57:46.994211 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-abf2097d-dddd5ff6-cq6ct" Jul 09 15:57:46.994577 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:57:46.994242 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-abf2097d-dddd5ff6-cq6ct" event={"ID":"a1203443-2b22-4887-90be-5d4509a4626b","Type":"ContainerDied","Data":"aecafefe701156295c8b75044d857fdf50b5b989b1fccdfeb9ddd274120409fb"} Jul 09 15:57:46.994577 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:57:46.994294 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-abf2097d-dddd5ff6-cq6ct" event={"ID":"a1203443-2b22-4887-90be-5d4509a4626b","Type":"ContainerDied","Data":"4563876fe653918df855f57d3f3cae75e6843da914ec9775347d77e94fcc3223"} Jul 09 15:57:46.994577 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:57:46.994318 2573 scope.go:117] "RemoveContainer" containerID="aecafefe701156295c8b75044d857fdf50b5b989b1fccdfeb9ddd274120409fb" Jul 09 15:57:47.003037 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:57:47.003017 2573 scope.go:117] "RemoveContainer" containerID="aecafefe701156295c8b75044d857fdf50b5b989b1fccdfeb9ddd274120409fb" Jul 09 15:57:47.003281 ip-10-0-133-183 kubenswrapper[2573]: E0709 15:57:47.003265 2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"aecafefe701156295c8b75044d857fdf50b5b989b1fccdfeb9ddd274120409fb\": container with ID starting with aecafefe701156295c8b75044d857fdf50b5b989b1fccdfeb9ddd274120409fb not found: ID does not exist" containerID="aecafefe701156295c8b75044d857fdf50b5b989b1fccdfeb9ddd274120409fb" Jul 09 15:57:47.003321 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:57:47.003290 2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"aecafefe701156295c8b75044d857fdf50b5b989b1fccdfeb9ddd274120409fb"} err="failed to get container status \"aecafefe701156295c8b75044d857fdf50b5b989b1fccdfeb9ddd274120409fb\": rpc error: code = NotFound desc = could not find container \"aecafefe701156295c8b75044d857fdf50b5b989b1fccdfeb9ddd274120409fb\": container with ID starting with aecafefe701156295c8b75044d857fdf50b5b989b1fccdfeb9ddd274120409fb not found: ID does not exist" Jul 09 15:57:47.018121 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:57:47.018093 2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-abf2097d-dddd5ff6-cq6ct"] Jul 09 15:57:47.026776 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:57:47.026754 2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-abf2097d-dddd5ff6-cq6ct"] Jul 09 15:57:48.811039 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:57:48.811009 2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a1203443-2b22-4887-90be-5d4509a4626b" path="/var/lib/kubelet/pods/a1203443-2b22-4887-90be-5d4509a4626b/volumes" Jul 09 15:57:59.430686 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:57:59.430603 2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-545edf81-565b96b49d-xjnr7"] Jul 09 15:57:59.431189 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:57:59.431171 2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a1203443-2b22-4887-90be-5d4509a4626b" containerName="maas-api" Jul 09 15:57:59.431256 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:57:59.431191 2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="a1203443-2b22-4887-90be-5d4509a4626b" containerName="maas-api" Jul 09 15:57:59.431305 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:57:59.431292 2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="a1203443-2b22-4887-90be-5d4509a4626b" containerName="maas-api" Jul 09 15:57:59.434852 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:57:59.434831 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-545edf81-565b96b49d-xjnr7" Jul 09 15:57:59.437799 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:57:59.437777 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-derive-545edf81-serving-cert\"" Jul 09 15:57:59.444155 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:57:59.444129 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-545edf81-565b96b49d-xjnr7"] Jul 09 15:57:59.542755 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:57:59.542723 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/550402eb-6e5c-4609-9056-a5d21ce37106-maas-api-tls\") pod \"maas-api-e2e-derive-545edf81-565b96b49d-xjnr7\" (UID: \"550402eb-6e5c-4609-9056-a5d21ce37106\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-545edf81-565b96b49d-xjnr7" Jul 09 15:57:59.542964 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:57:59.542764 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/550402eb-6e5c-4609-9056-a5d21ce37106-openshift-service-ca\") pod \"maas-api-e2e-derive-545edf81-565b96b49d-xjnr7\" (UID: \"550402eb-6e5c-4609-9056-a5d21ce37106\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-545edf81-565b96b49d-xjnr7" Jul 09 15:57:59.542964 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:57:59.542903 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wflqd\" (UniqueName: \"kubernetes.io/projected/550402eb-6e5c-4609-9056-a5d21ce37106-kube-api-access-wflqd\") pod \"maas-api-e2e-derive-545edf81-565b96b49d-xjnr7\" (UID: \"550402eb-6e5c-4609-9056-a5d21ce37106\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-545edf81-565b96b49d-xjnr7" Jul 09 15:57:59.644350 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:57:59.644314 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wflqd\" (UniqueName: \"kubernetes.io/projected/550402eb-6e5c-4609-9056-a5d21ce37106-kube-api-access-wflqd\") pod \"maas-api-e2e-derive-545edf81-565b96b49d-xjnr7\" (UID: \"550402eb-6e5c-4609-9056-a5d21ce37106\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-545edf81-565b96b49d-xjnr7" Jul 09 15:57:59.644524 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:57:59.644365 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/550402eb-6e5c-4609-9056-a5d21ce37106-maas-api-tls\") pod \"maas-api-e2e-derive-545edf81-565b96b49d-xjnr7\" (UID: \"550402eb-6e5c-4609-9056-a5d21ce37106\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-545edf81-565b96b49d-xjnr7" Jul 09 15:57:59.644524 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:57:59.644395 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/550402eb-6e5c-4609-9056-a5d21ce37106-openshift-service-ca\") pod \"maas-api-e2e-derive-545edf81-565b96b49d-xjnr7\" (UID: \"550402eb-6e5c-4609-9056-a5d21ce37106\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-545edf81-565b96b49d-xjnr7" Jul 09 15:57:59.644524 ip-10-0-133-183 kubenswrapper[2573]: E0709 15:57:59.644494 2573 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-derive-545edf81-serving-cert: secret "maas-api-e2e-derive-545edf81-serving-cert" not found Jul 09 15:57:59.644652 ip-10-0-133-183 kubenswrapper[2573]: E0709 15:57:59.644572 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/550402eb-6e5c-4609-9056-a5d21ce37106-maas-api-tls podName:550402eb-6e5c-4609-9056-a5d21ce37106 nodeName:}" failed. No retries permitted until 2026-07-09 15:58:00.144553311 +0000 UTC m=+1601.866060178 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/550402eb-6e5c-4609-9056-a5d21ce37106-maas-api-tls") pod "maas-api-e2e-derive-545edf81-565b96b49d-xjnr7" (UID: "550402eb-6e5c-4609-9056-a5d21ce37106") : secret "maas-api-e2e-derive-545edf81-serving-cert" not found Jul 09 15:57:59.645142 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:57:59.645123 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/550402eb-6e5c-4609-9056-a5d21ce37106-openshift-service-ca\") pod \"maas-api-e2e-derive-545edf81-565b96b49d-xjnr7\" (UID: \"550402eb-6e5c-4609-9056-a5d21ce37106\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-545edf81-565b96b49d-xjnr7" Jul 09 15:57:59.653263 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:57:59.653240 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wflqd\" (UniqueName: \"kubernetes.io/projected/550402eb-6e5c-4609-9056-a5d21ce37106-kube-api-access-wflqd\") pod \"maas-api-e2e-derive-545edf81-565b96b49d-xjnr7\" (UID: \"550402eb-6e5c-4609-9056-a5d21ce37106\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-545edf81-565b96b49d-xjnr7" Jul 09 15:58:00.148868 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:58:00.148829 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/550402eb-6e5c-4609-9056-a5d21ce37106-maas-api-tls\") pod \"maas-api-e2e-derive-545edf81-565b96b49d-xjnr7\" (UID: \"550402eb-6e5c-4609-9056-a5d21ce37106\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-545edf81-565b96b49d-xjnr7" Jul 09 15:58:00.151138 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:58:00.151118 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/550402eb-6e5c-4609-9056-a5d21ce37106-maas-api-tls\") pod \"maas-api-e2e-derive-545edf81-565b96b49d-xjnr7\" (UID: \"550402eb-6e5c-4609-9056-a5d21ce37106\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-545edf81-565b96b49d-xjnr7" Jul 09 15:58:00.346965 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:58:00.346926 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-545edf81-565b96b49d-xjnr7" Jul 09 15:58:00.487277 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:58:00.487254 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-545edf81-565b96b49d-xjnr7"] Jul 09 15:58:00.489912 ip-10-0-133-183 kubenswrapper[2573]: W0709 15:58:00.489856 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod550402eb_6e5c_4609_9056_a5d21ce37106.slice/crio-92cdd574f450413625dbd16deaf1e698c1fd90ce3ef36006d1bc6a8b3aba0adf WatchSource:0}: Error finding container 92cdd574f450413625dbd16deaf1e698c1fd90ce3ef36006d1bc6a8b3aba0adf: Status 404 returned error can't find the container with id 92cdd574f450413625dbd16deaf1e698c1fd90ce3ef36006d1bc6a8b3aba0adf Jul 09 15:58:01.064105 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:58:01.064063 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-545edf81-565b96b49d-xjnr7" event={"ID":"550402eb-6e5c-4609-9056-a5d21ce37106","Type":"ContainerStarted","Data":"10b2ef854681db2d7b22a553018f42bd817e46cbddb0b6fc85557071d16ab847"} Jul 09 15:58:01.064302 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:58:01.064131 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-545edf81-565b96b49d-xjnr7" event={"ID":"550402eb-6e5c-4609-9056-a5d21ce37106","Type":"ContainerStarted","Data":"92cdd574f450413625dbd16deaf1e698c1fd90ce3ef36006d1bc6a8b3aba0adf"} Jul 09 15:58:01.064302 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:58:01.064217 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-derive-545edf81-565b96b49d-xjnr7" Jul 09 15:58:01.083468 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:58:01.083416 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-derive-545edf81-565b96b49d-xjnr7" podStartSLOduration=1.636335811 podStartE2EDuration="2.083400548s" totalImagesPullingTime="447.064737ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 15:57:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 15:58:01.081145963 +0000 UTC m=+1602.802652822" watchObservedRunningTime="2026-07-09 15:58:01.083400548 +0000 UTC m=+1602.804907415" Jul 09 15:58:04.229664 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:58:04.229623 2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-545edf81-565b96b49d-xjnr7"] Jul 09 15:58:04.230111 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:58:04.229856 2573 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-derive-545edf81-565b96b49d-xjnr7" podUID="550402eb-6e5c-4609-9056-a5d21ce37106" containerName="maas-api" containerID="cri-o://10b2ef854681db2d7b22a553018f42bd817e46cbddb0b6fc85557071d16ab847" gracePeriod=30 Jul 09 15:58:04.483136 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:58:04.483074 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-545edf81-565b96b49d-xjnr7" Jul 09 15:58:04.587615 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:58:04.587576 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/550402eb-6e5c-4609-9056-a5d21ce37106-maas-api-tls\") pod \"550402eb-6e5c-4609-9056-a5d21ce37106\" (UID: \"550402eb-6e5c-4609-9056-a5d21ce37106\") " Jul 09 15:58:04.587806 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:58:04.587663 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wflqd\" (UniqueName: \"kubernetes.io/projected/550402eb-6e5c-4609-9056-a5d21ce37106-kube-api-access-wflqd\") pod \"550402eb-6e5c-4609-9056-a5d21ce37106\" (UID: \"550402eb-6e5c-4609-9056-a5d21ce37106\") " Jul 09 15:58:04.587806 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:58:04.587720 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/550402eb-6e5c-4609-9056-a5d21ce37106-openshift-service-ca\") pod \"550402eb-6e5c-4609-9056-a5d21ce37106\" (UID: \"550402eb-6e5c-4609-9056-a5d21ce37106\") " Jul 09 15:58:04.588165 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:58:04.588135 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/550402eb-6e5c-4609-9056-a5d21ce37106-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "550402eb-6e5c-4609-9056-a5d21ce37106" (UID: "550402eb-6e5c-4609-9056-a5d21ce37106"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 09 15:58:04.589760 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:58:04.589738 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/550402eb-6e5c-4609-9056-a5d21ce37106-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "550402eb-6e5c-4609-9056-a5d21ce37106" (UID: "550402eb-6e5c-4609-9056-a5d21ce37106"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 09 15:58:04.589857 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:58:04.589768 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/550402eb-6e5c-4609-9056-a5d21ce37106-kube-api-access-wflqd" (OuterVolumeSpecName: "kube-api-access-wflqd") pod "550402eb-6e5c-4609-9056-a5d21ce37106" (UID: "550402eb-6e5c-4609-9056-a5d21ce37106"). InnerVolumeSpecName "kube-api-access-wflqd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 09 15:58:04.688916 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:58:04.688859 2573 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/550402eb-6e5c-4609-9056-a5d21ce37106-openshift-service-ca\") on node \"ip-10-0-133-183.ec2.internal\" DevicePath \"\"" Jul 09 15:58:04.688916 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:58:04.688907 2573 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/550402eb-6e5c-4609-9056-a5d21ce37106-maas-api-tls\") on node \"ip-10-0-133-183.ec2.internal\" DevicePath \"\"" Jul 09 15:58:04.688916 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:58:04.688920 2573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wflqd\" (UniqueName: \"kubernetes.io/projected/550402eb-6e5c-4609-9056-a5d21ce37106-kube-api-access-wflqd\") on node \"ip-10-0-133-183.ec2.internal\" DevicePath \"\"" Jul 09 15:58:05.083437 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:58:05.083345 2573 generic.go:356] "Generic (PLEG): container finished" podID="550402eb-6e5c-4609-9056-a5d21ce37106" containerID="10b2ef854681db2d7b22a553018f42bd817e46cbddb0b6fc85557071d16ab847" exitCode=0 Jul 09 15:58:05.083437 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:58:05.083387 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-545edf81-565b96b49d-xjnr7" Jul 09 15:58:05.083695 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:58:05.083435 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-545edf81-565b96b49d-xjnr7" event={"ID":"550402eb-6e5c-4609-9056-a5d21ce37106","Type":"ContainerDied","Data":"10b2ef854681db2d7b22a553018f42bd817e46cbddb0b6fc85557071d16ab847"} Jul 09 15:58:05.083695 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:58:05.083486 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-545edf81-565b96b49d-xjnr7" event={"ID":"550402eb-6e5c-4609-9056-a5d21ce37106","Type":"ContainerDied","Data":"92cdd574f450413625dbd16deaf1e698c1fd90ce3ef36006d1bc6a8b3aba0adf"} Jul 09 15:58:05.083695 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:58:05.083509 2573 scope.go:117] "RemoveContainer" containerID="10b2ef854681db2d7b22a553018f42bd817e46cbddb0b6fc85557071d16ab847" Jul 09 15:58:05.092383 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:58:05.092363 2573 scope.go:117] "RemoveContainer" containerID="10b2ef854681db2d7b22a553018f42bd817e46cbddb0b6fc85557071d16ab847" Jul 09 15:58:05.092669 ip-10-0-133-183 kubenswrapper[2573]: E0709 15:58:05.092650 2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"10b2ef854681db2d7b22a553018f42bd817e46cbddb0b6fc85557071d16ab847\": container with ID starting with 10b2ef854681db2d7b22a553018f42bd817e46cbddb0b6fc85557071d16ab847 not found: ID does not exist" containerID="10b2ef854681db2d7b22a553018f42bd817e46cbddb0b6fc85557071d16ab847" Jul 09 15:58:05.092717 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:58:05.092679 2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"10b2ef854681db2d7b22a553018f42bd817e46cbddb0b6fc85557071d16ab847"} err="failed to get container status \"10b2ef854681db2d7b22a553018f42bd817e46cbddb0b6fc85557071d16ab847\": rpc error: code = NotFound desc = could not find container \"10b2ef854681db2d7b22a553018f42bd817e46cbddb0b6fc85557071d16ab847\": container with ID starting with 10b2ef854681db2d7b22a553018f42bd817e46cbddb0b6fc85557071d16ab847 not found: ID does not exist" Jul 09 15:58:05.102359 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:58:05.102333 2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-545edf81-565b96b49d-xjnr7"] Jul 09 15:58:05.109809 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:58:05.109787 2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-545edf81-565b96b49d-xjnr7"] Jul 09 15:58:06.811787 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:58:06.811748 2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="550402eb-6e5c-4609-9056-a5d21ce37106" path="/var/lib/kubelet/pods/550402eb-6e5c-4609-9056-a5d21ce37106/volumes" Jul 09 15:59:51.640928 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:59:51.640823 2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-4d56b236-75c65684b8-dm7w4"] Jul 09 15:59:51.641413 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:59:51.641383 2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="550402eb-6e5c-4609-9056-a5d21ce37106" containerName="maas-api" Jul 09 15:59:51.641413 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:59:51.641398 2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="550402eb-6e5c-4609-9056-a5d21ce37106" containerName="maas-api" Jul 09 15:59:51.641516 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:59:51.641503 2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="550402eb-6e5c-4609-9056-a5d21ce37106" containerName="maas-api" Jul 09 15:59:51.644902 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:59:51.644870 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-4d56b236-75c65684b8-dm7w4" Jul 09 15:59:51.647332 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:59:51.647310 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-mt-4d56b236-serving-cert\"" Jul 09 15:59:51.653719 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:59:51.653631 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-4d56b236-75c65684b8-dm7w4"] Jul 09 15:59:51.671463 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:59:51.671434 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9bdef074-66c2-4e52-b93f-f65cb7354b43-openshift-service-ca\") pod \"maas-api-e2e-mt-4d56b236-75c65684b8-dm7w4\" (UID: \"9bdef074-66c2-4e52-b93f-f65cb7354b43\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-4d56b236-75c65684b8-dm7w4" Jul 09 15:59:51.671654 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:59:51.671483 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wn96r\" (UniqueName: \"kubernetes.io/projected/9bdef074-66c2-4e52-b93f-f65cb7354b43-kube-api-access-wn96r\") pod \"maas-api-e2e-mt-4d56b236-75c65684b8-dm7w4\" (UID: \"9bdef074-66c2-4e52-b93f-f65cb7354b43\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-4d56b236-75c65684b8-dm7w4" Jul 09 15:59:51.671654 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:59:51.671510 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9bdef074-66c2-4e52-b93f-f65cb7354b43-maas-api-tls\") pod \"maas-api-e2e-mt-4d56b236-75c65684b8-dm7w4\" (UID: \"9bdef074-66c2-4e52-b93f-f65cb7354b43\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-4d56b236-75c65684b8-dm7w4" Jul 09 15:59:51.772622 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:59:51.772582 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9bdef074-66c2-4e52-b93f-f65cb7354b43-openshift-service-ca\") pod \"maas-api-e2e-mt-4d56b236-75c65684b8-dm7w4\" (UID: \"9bdef074-66c2-4e52-b93f-f65cb7354b43\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-4d56b236-75c65684b8-dm7w4" Jul 09 15:59:51.772802 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:59:51.772639 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wn96r\" (UniqueName: \"kubernetes.io/projected/9bdef074-66c2-4e52-b93f-f65cb7354b43-kube-api-access-wn96r\") pod \"maas-api-e2e-mt-4d56b236-75c65684b8-dm7w4\" (UID: \"9bdef074-66c2-4e52-b93f-f65cb7354b43\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-4d56b236-75c65684b8-dm7w4" Jul 09 15:59:51.772802 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:59:51.772669 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9bdef074-66c2-4e52-b93f-f65cb7354b43-maas-api-tls\") pod \"maas-api-e2e-mt-4d56b236-75c65684b8-dm7w4\" (UID: \"9bdef074-66c2-4e52-b93f-f65cb7354b43\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-4d56b236-75c65684b8-dm7w4" Jul 09 15:59:51.772802 ip-10-0-133-183 kubenswrapper[2573]: E0709 15:59:51.772771 2573 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-mt-4d56b236-serving-cert: secret "maas-api-e2e-mt-4d56b236-serving-cert" not found Jul 09 15:59:51.772954 ip-10-0-133-183 kubenswrapper[2573]: E0709 15:59:51.772832 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9bdef074-66c2-4e52-b93f-f65cb7354b43-maas-api-tls podName:9bdef074-66c2-4e52-b93f-f65cb7354b43 nodeName:}" failed. No retries permitted until 2026-07-09 15:59:52.272816603 +0000 UTC m=+1713.994323449 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/9bdef074-66c2-4e52-b93f-f65cb7354b43-maas-api-tls") pod "maas-api-e2e-mt-4d56b236-75c65684b8-dm7w4" (UID: "9bdef074-66c2-4e52-b93f-f65cb7354b43") : secret "maas-api-e2e-mt-4d56b236-serving-cert" not found Jul 09 15:59:51.773319 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:59:51.773298 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9bdef074-66c2-4e52-b93f-f65cb7354b43-openshift-service-ca\") pod \"maas-api-e2e-mt-4d56b236-75c65684b8-dm7w4\" (UID: \"9bdef074-66c2-4e52-b93f-f65cb7354b43\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-4d56b236-75c65684b8-dm7w4" Jul 09 15:59:51.784279 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:59:51.784254 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wn96r\" (UniqueName: \"kubernetes.io/projected/9bdef074-66c2-4e52-b93f-f65cb7354b43-kube-api-access-wn96r\") pod \"maas-api-e2e-mt-4d56b236-75c65684b8-dm7w4\" (UID: \"9bdef074-66c2-4e52-b93f-f65cb7354b43\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-4d56b236-75c65684b8-dm7w4" Jul 09 15:59:52.277838 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:59:52.277806 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9bdef074-66c2-4e52-b93f-f65cb7354b43-maas-api-tls\") pod \"maas-api-e2e-mt-4d56b236-75c65684b8-dm7w4\" (UID: \"9bdef074-66c2-4e52-b93f-f65cb7354b43\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-4d56b236-75c65684b8-dm7w4" Jul 09 15:59:52.280232 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:59:52.280208 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9bdef074-66c2-4e52-b93f-f65cb7354b43-maas-api-tls\") pod \"maas-api-e2e-mt-4d56b236-75c65684b8-dm7w4\" (UID: \"9bdef074-66c2-4e52-b93f-f65cb7354b43\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-4d56b236-75c65684b8-dm7w4" Jul 09 15:59:52.558048 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:59:52.557960 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-4d56b236-75c65684b8-dm7w4" Jul 09 15:59:52.700517 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:59:52.700497 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-4d56b236-75c65684b8-dm7w4"] Jul 09 15:59:52.703304 ip-10-0-133-183 kubenswrapper[2573]: W0709 15:59:52.703275 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9bdef074_66c2_4e52_b93f_f65cb7354b43.slice/crio-c14d5159d0052e6bb6fc00dc000dc5a7801be3a7749a281cbb652ea72730c605 WatchSource:0}: Error finding container c14d5159d0052e6bb6fc00dc000dc5a7801be3a7749a281cbb652ea72730c605: Status 404 returned error can't find the container with id c14d5159d0052e6bb6fc00dc000dc5a7801be3a7749a281cbb652ea72730c605 Jul 09 15:59:53.587682 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:59:53.587619 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-4d56b236-75c65684b8-dm7w4" event={"ID":"9bdef074-66c2-4e52-b93f-f65cb7354b43","Type":"ContainerStarted","Data":"2115070e486c1eeab81595235df09e65ba90c79ada308e5a1c5440c290fda9b6"} Jul 09 15:59:53.587866 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:59:53.587726 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-4d56b236-75c65684b8-dm7w4" event={"ID":"9bdef074-66c2-4e52-b93f-f65cb7354b43","Type":"ContainerStarted","Data":"c14d5159d0052e6bb6fc00dc000dc5a7801be3a7749a281cbb652ea72730c605"} Jul 09 15:59:53.587866 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:59:53.587757 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-4d56b236-75c65684b8-dm7w4" Jul 09 15:59:53.608752 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:59:53.608699 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-mt-4d56b236-75c65684b8-dm7w4" podStartSLOduration=2.189326853 podStartE2EDuration="2.608683505s" totalImagesPullingTime="419.356652ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 15:59:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 15:59:53.606340272 +0000 UTC m=+1715.327847185" watchObservedRunningTime="2026-07-09 15:59:53.608683505 +0000 UTC m=+1715.330190375" Jul 09 15:59:59.597825 ip-10-0-133-183 kubenswrapper[2573]: I0709 15:59:59.597796 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-4d56b236-75c65684b8-dm7w4" Jul 09 16:00:00.133790 ip-10-0-133-183 kubenswrapper[2573]: I0709 16:00:00.133745 2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29726880-mwwxt"] Jul 09 16:00:00.137818 ip-10-0-133-183 kubenswrapper[2573]: I0709 16:00:00.137800 2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-4d56b236-29726880-4wrdx"] Jul 09 16:00:00.137964 ip-10-0-133-183 kubenswrapper[2573]: I0709 16:00:00.137945 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29726880-mwwxt" Jul 09 16:00:00.141449 ip-10-0-133-183 kubenswrapper[2573]: I0709 16:00:00.141290 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-4d56b236-29726880-4wrdx" Jul 09 16:00:00.143849 ip-10-0-133-183 kubenswrapper[2573]: I0709 16:00:00.143828 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29726880-mwwxt"] Jul 09 16:00:00.150464 ip-10-0-133-183 kubenswrapper[2573]: I0709 16:00:00.150441 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29726880-mwwxt" Jul 09 16:00:00.158134 ip-10-0-133-183 kubenswrapper[2573]: I0709 16:00:00.158111 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-4d56b236-29726880-4wrdx"] Jul 09 16:00:00.159257 ip-10-0-133-183 kubenswrapper[2573]: I0709 16:00:00.159088 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-4d56b236-29726880-4wrdx" Jul 09 16:00:00.302821 ip-10-0-133-183 kubenswrapper[2573]: I0709 16:00:00.302768 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29726880-mwwxt"] Jul 09 16:00:00.305480 ip-10-0-133-183 kubenswrapper[2573]: W0709 16:00:00.305454 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8cbe8ee2_5106_4a49_8035_cf751d58496d.slice/crio-dc3408af7729cce099410f36aafa567cb874858b24ac287529985ac7b0652387 WatchSource:0}: Error finding container dc3408af7729cce099410f36aafa567cb874858b24ac287529985ac7b0652387: Status 404 returned error can't find the container with id dc3408af7729cce099410f36aafa567cb874858b24ac287529985ac7b0652387 Jul 09 16:00:00.328592 ip-10-0-133-183 kubenswrapper[2573]: I0709 16:00:00.328561 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-4d56b236-29726880-4wrdx"] Jul 09 16:00:00.332185 ip-10-0-133-183 kubenswrapper[2573]: W0709 16:00:00.332157 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd3bb0b8b_0bd1_4572_89aa_7c105fde52ab.slice/crio-345e6748ebb9d9eba1125ba67911239c530fc903742feaec704809213c09eaba WatchSource:0}: Error finding container 345e6748ebb9d9eba1125ba67911239c530fc903742feaec704809213c09eaba: Status 404 returned error can't find the container with id 345e6748ebb9d9eba1125ba67911239c530fc903742feaec704809213c09eaba Jul 09 16:00:00.622861 ip-10-0-133-183 kubenswrapper[2573]: I0709 16:00:00.622823 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-4d56b236-29726880-4wrdx" event={"ID":"d3bb0b8b-0bd1-4572-89aa-7c105fde52ab","Type":"ContainerStarted","Data":"345e6748ebb9d9eba1125ba67911239c530fc903742feaec704809213c09eaba"} Jul 09 16:00:00.623900 ip-10-0-133-183 kubenswrapper[2573]: I0709 16:00:00.623858 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29726880-mwwxt" event={"ID":"8cbe8ee2-5106-4a49-8035-cf751d58496d","Type":"ContainerStarted","Data":"dc3408af7729cce099410f36aafa567cb874858b24ac287529985ac7b0652387"} Jul 09 16:00:03.642485 ip-10-0-133-183 kubenswrapper[2573]: I0709 16:00:03.642446 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29726880-mwwxt" event={"ID":"8cbe8ee2-5106-4a49-8035-cf751d58496d","Type":"ContainerStarted","Data":"57e1fb0023ec09bdd4fbf8fc58f62716515f71081387b41160425a8ffd226910"} Jul 09 16:00:03.643993 ip-10-0-133-183 kubenswrapper[2573]: I0709 16:00:03.643966 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-4d56b236-29726880-4wrdx" event={"ID":"d3bb0b8b-0bd1-4572-89aa-7c105fde52ab","Type":"ContainerStarted","Data":"0820c15392bd5df1d74f01227bae3d973d07ada2192aa41c797ca98ea7903c25"} Jul 09 16:00:03.660249 ip-10-0-133-183 kubenswrapper[2573]: I0709 16:00:03.660198 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29726880-mwwxt" podStartSLOduration=0.782023541 podStartE2EDuration="3.660180635s" totalImagesPullingTime="2.878157094s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 16:00:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 16:00:03.65847912 +0000 UTC m=+1725.379985990" watchObservedRunningTime="2026-07-09 16:00:03.660180635 +0000 UTC m=+1725.381687503" Jul 09 16:00:03.676011 ip-10-0-133-183 kubenswrapper[2573]: I0709 16:00:03.675963 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-4d56b236-29726880-4wrdx" podStartSLOduration=0.823402544 podStartE2EDuration="3.675946463s" totalImagesPullingTime="2.852543919s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 16:00:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 16:00:03.675105417 +0000 UTC m=+1725.396612304" watchObservedRunningTime="2026-07-09 16:00:03.675946463 +0000 UTC m=+1725.397453332" Jul 09 16:00:04.226536 ip-10-0-133-183 kubenswrapper[2573]: I0709 16:00:04.226504 2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-4d56b236-75c65684b8-dm7w4"] Jul 09 16:00:04.226735 ip-10-0-133-183 kubenswrapper[2573]: I0709 16:00:04.226712 2573 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-mt-4d56b236-75c65684b8-dm7w4" podUID="9bdef074-66c2-4e52-b93f-f65cb7354b43" containerName="maas-api" containerID="cri-o://2115070e486c1eeab81595235df09e65ba90c79ada308e5a1c5440c290fda9b6" gracePeriod=30 Jul 09 16:00:04.251555 ip-10-0-133-183 kubenswrapper[2573]: I0709 16:00:04.251528 2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-4d56b236-29726880-4wrdx"] Jul 09 16:00:04.482705 ip-10-0-133-183 kubenswrapper[2573]: I0709 16:00:04.482639 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-4d56b236-75c65684b8-dm7w4" Jul 09 16:00:04.596320 ip-10-0-133-183 kubenswrapper[2573]: I0709 16:00:04.596284 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9bdef074-66c2-4e52-b93f-f65cb7354b43-openshift-service-ca\") pod \"9bdef074-66c2-4e52-b93f-f65cb7354b43\" (UID: \"9bdef074-66c2-4e52-b93f-f65cb7354b43\") " Jul 09 16:00:04.596320 ip-10-0-133-183 kubenswrapper[2573]: I0709 16:00:04.596336 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9bdef074-66c2-4e52-b93f-f65cb7354b43-maas-api-tls\") pod \"9bdef074-66c2-4e52-b93f-f65cb7354b43\" (UID: \"9bdef074-66c2-4e52-b93f-f65cb7354b43\") " Jul 09 16:00:04.596578 ip-10-0-133-183 kubenswrapper[2573]: I0709 16:00:04.596396 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wn96r\" (UniqueName: \"kubernetes.io/projected/9bdef074-66c2-4e52-b93f-f65cb7354b43-kube-api-access-wn96r\") pod \"9bdef074-66c2-4e52-b93f-f65cb7354b43\" (UID: \"9bdef074-66c2-4e52-b93f-f65cb7354b43\") " Jul 09 16:00:04.596672 ip-10-0-133-183 kubenswrapper[2573]: I0709 16:00:04.596644 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9bdef074-66c2-4e52-b93f-f65cb7354b43-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "9bdef074-66c2-4e52-b93f-f65cb7354b43" (UID: "9bdef074-66c2-4e52-b93f-f65cb7354b43"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 09 16:00:04.598520 ip-10-0-133-183 kubenswrapper[2573]: I0709 16:00:04.598483 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9bdef074-66c2-4e52-b93f-f65cb7354b43-kube-api-access-wn96r" (OuterVolumeSpecName: "kube-api-access-wn96r") pod "9bdef074-66c2-4e52-b93f-f65cb7354b43" (UID: "9bdef074-66c2-4e52-b93f-f65cb7354b43"). InnerVolumeSpecName "kube-api-access-wn96r". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 09 16:00:04.598520 ip-10-0-133-183 kubenswrapper[2573]: I0709 16:00:04.598508 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9bdef074-66c2-4e52-b93f-f65cb7354b43-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "9bdef074-66c2-4e52-b93f-f65cb7354b43" (UID: "9bdef074-66c2-4e52-b93f-f65cb7354b43"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 09 16:00:04.650287 ip-10-0-133-183 kubenswrapper[2573]: I0709 16:00:04.650253 2573 generic.go:356] "Generic (PLEG): container finished" podID="9bdef074-66c2-4e52-b93f-f65cb7354b43" containerID="2115070e486c1eeab81595235df09e65ba90c79ada308e5a1c5440c290fda9b6" exitCode=0 Jul 09 16:00:04.650744 ip-10-0-133-183 kubenswrapper[2573]: I0709 16:00:04.650291 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-4d56b236-75c65684b8-dm7w4" event={"ID":"9bdef074-66c2-4e52-b93f-f65cb7354b43","Type":"ContainerDied","Data":"2115070e486c1eeab81595235df09e65ba90c79ada308e5a1c5440c290fda9b6"} Jul 09 16:00:04.650744 ip-10-0-133-183 kubenswrapper[2573]: I0709 16:00:04.650332 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-4d56b236-75c65684b8-dm7w4" Jul 09 16:00:04.650744 ip-10-0-133-183 kubenswrapper[2573]: I0709 16:00:04.650393 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-4d56b236-75c65684b8-dm7w4" event={"ID":"9bdef074-66c2-4e52-b93f-f65cb7354b43","Type":"ContainerDied","Data":"c14d5159d0052e6bb6fc00dc000dc5a7801be3a7749a281cbb652ea72730c605"} Jul 09 16:00:04.650744 ip-10-0-133-183 kubenswrapper[2573]: I0709 16:00:04.650426 2573 scope.go:117] "RemoveContainer" containerID="2115070e486c1eeab81595235df09e65ba90c79ada308e5a1c5440c290fda9b6" Jul 09 16:00:04.659836 ip-10-0-133-183 kubenswrapper[2573]: I0709 16:00:04.659811 2573 scope.go:117] "RemoveContainer" containerID="2115070e486c1eeab81595235df09e65ba90c79ada308e5a1c5440c290fda9b6" Jul 09 16:00:04.660154 ip-10-0-133-183 kubenswrapper[2573]: E0709 16:00:04.660121 2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2115070e486c1eeab81595235df09e65ba90c79ada308e5a1c5440c290fda9b6\": container with ID starting with 2115070e486c1eeab81595235df09e65ba90c79ada308e5a1c5440c290fda9b6 not found: ID does not exist" containerID="2115070e486c1eeab81595235df09e65ba90c79ada308e5a1c5440c290fda9b6" Jul 09 16:00:04.660244 ip-10-0-133-183 kubenswrapper[2573]: I0709 16:00:04.660164 2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2115070e486c1eeab81595235df09e65ba90c79ada308e5a1c5440c290fda9b6"} err="failed to get container status \"2115070e486c1eeab81595235df09e65ba90c79ada308e5a1c5440c290fda9b6\": rpc error: code = NotFound desc = could not find container \"2115070e486c1eeab81595235df09e65ba90c79ada308e5a1c5440c290fda9b6\": container with ID starting with 2115070e486c1eeab81595235df09e65ba90c79ada308e5a1c5440c290fda9b6 not found: ID does not exist" Jul 09 16:00:04.680716 ip-10-0-133-183 kubenswrapper[2573]: I0709 16:00:04.680660 2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-4d56b236-75c65684b8-dm7w4"] Jul 09 16:00:04.683159 ip-10-0-133-183 kubenswrapper[2573]: I0709 16:00:04.683130 2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-4d56b236-75c65684b8-dm7w4"] Jul 09 16:00:04.697817 ip-10-0-133-183 kubenswrapper[2573]: I0709 16:00:04.697793 2573 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9bdef074-66c2-4e52-b93f-f65cb7354b43-openshift-service-ca\") on node \"ip-10-0-133-183.ec2.internal\" DevicePath \"\"" Jul 09 16:00:04.697817 ip-10-0-133-183 kubenswrapper[2573]: I0709 16:00:04.697817 2573 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9bdef074-66c2-4e52-b93f-f65cb7354b43-maas-api-tls\") on node \"ip-10-0-133-183.ec2.internal\" DevicePath \"\"" Jul 09 16:00:04.698000 ip-10-0-133-183 kubenswrapper[2573]: I0709 16:00:04.697829 2573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wn96r\" (UniqueName: \"kubernetes.io/projected/9bdef074-66c2-4e52-b93f-f65cb7354b43-kube-api-access-wn96r\") on node \"ip-10-0-133-183.ec2.internal\" DevicePath \"\"" Jul 09 16:00:04.811355 ip-10-0-133-183 kubenswrapper[2573]: I0709 16:00:04.811271 2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9bdef074-66c2-4e52-b93f-f65cb7354b43" path="/var/lib/kubelet/pods/9bdef074-66c2-4e52-b93f-f65cb7354b43/volumes" Jul 09 16:00:05.656358 ip-10-0-133-183 kubenswrapper[2573]: I0709 16:00:05.656318 2573 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-4d56b236-29726880-4wrdx" podUID="d3bb0b8b-0bd1-4572-89aa-7c105fde52ab" containerName="cleanup" containerID="cri-o://0820c15392bd5df1d74f01227bae3d973d07ada2192aa41c797ca98ea7903c25" gracePeriod=30 Jul 09 16:00:34.797387 ip-10-0-133-183 kubenswrapper[2573]: I0709 16:00:34.797349 2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-378f7a5c-79dd54ffd6-77rt8"] Jul 09 16:00:34.798014 ip-10-0-133-183 kubenswrapper[2573]: I0709 16:00:34.797946 2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9bdef074-66c2-4e52-b93f-f65cb7354b43" containerName="maas-api" Jul 09 16:00:34.798014 ip-10-0-133-183 kubenswrapper[2573]: I0709 16:00:34.797963 2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="9bdef074-66c2-4e52-b93f-f65cb7354b43" containerName="maas-api" Jul 09 16:00:34.798128 ip-10-0-133-183 kubenswrapper[2573]: I0709 16:00:34.798071 2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="9bdef074-66c2-4e52-b93f-f65cb7354b43" containerName="maas-api" Jul 09 16:00:34.803110 ip-10-0-133-183 kubenswrapper[2573]: I0709 16:00:34.803089 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-378f7a5c-79dd54ffd6-77rt8" Jul 09 16:00:34.805622 ip-10-0-133-183 kubenswrapper[2573]: I0709 16:00:34.805593 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-mt-378f7a5c-serving-cert\"" Jul 09 16:00:34.813602 ip-10-0-133-183 kubenswrapper[2573]: I0709 16:00:34.813579 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-378f7a5c-79dd54ffd6-77rt8"] Jul 09 16:00:34.876711 ip-10-0-133-183 kubenswrapper[2573]: I0709 16:00:34.876666 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hp7lt\" (UniqueName: \"kubernetes.io/projected/c41bfea6-b89a-4572-9a27-8c517da2c352-kube-api-access-hp7lt\") pod \"maas-api-e2e-mt-378f7a5c-79dd54ffd6-77rt8\" (UID: \"c41bfea6-b89a-4572-9a27-8c517da2c352\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-378f7a5c-79dd54ffd6-77rt8" Jul 09 16:00:34.876941 ip-10-0-133-183 kubenswrapper[2573]: I0709 16:00:34.876780 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c41bfea6-b89a-4572-9a27-8c517da2c352-openshift-service-ca\") pod \"maas-api-e2e-mt-378f7a5c-79dd54ffd6-77rt8\" (UID: \"c41bfea6-b89a-4572-9a27-8c517da2c352\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-378f7a5c-79dd54ffd6-77rt8" Jul 09 16:00:34.876941 ip-10-0-133-183 kubenswrapper[2573]: I0709 16:00:34.876858 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c41bfea6-b89a-4572-9a27-8c517da2c352-maas-api-tls\") pod \"maas-api-e2e-mt-378f7a5c-79dd54ffd6-77rt8\" (UID: \"c41bfea6-b89a-4572-9a27-8c517da2c352\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-378f7a5c-79dd54ffd6-77rt8" Jul 09 16:00:34.978003 ip-10-0-133-183 kubenswrapper[2573]: I0709 16:00:34.977962 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c41bfea6-b89a-4572-9a27-8c517da2c352-openshift-service-ca\") pod \"maas-api-e2e-mt-378f7a5c-79dd54ffd6-77rt8\" (UID: \"c41bfea6-b89a-4572-9a27-8c517da2c352\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-378f7a5c-79dd54ffd6-77rt8" Jul 09 16:00:34.978190 ip-10-0-133-183 kubenswrapper[2573]: I0709 16:00:34.978020 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c41bfea6-b89a-4572-9a27-8c517da2c352-maas-api-tls\") pod \"maas-api-e2e-mt-378f7a5c-79dd54ffd6-77rt8\" (UID: \"c41bfea6-b89a-4572-9a27-8c517da2c352\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-378f7a5c-79dd54ffd6-77rt8" Jul 09 16:00:34.978190 ip-10-0-133-183 kubenswrapper[2573]: I0709 16:00:34.978047 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hp7lt\" (UniqueName: \"kubernetes.io/projected/c41bfea6-b89a-4572-9a27-8c517da2c352-kube-api-access-hp7lt\") pod \"maas-api-e2e-mt-378f7a5c-79dd54ffd6-77rt8\" (UID: \"c41bfea6-b89a-4572-9a27-8c517da2c352\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-378f7a5c-79dd54ffd6-77rt8" Jul 09 16:00:34.978627 ip-10-0-133-183 kubenswrapper[2573]: I0709 16:00:34.978602 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c41bfea6-b89a-4572-9a27-8c517da2c352-openshift-service-ca\") pod \"maas-api-e2e-mt-378f7a5c-79dd54ffd6-77rt8\" (UID: \"c41bfea6-b89a-4572-9a27-8c517da2c352\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-378f7a5c-79dd54ffd6-77rt8" Jul 09 16:00:34.980457 ip-10-0-133-183 kubenswrapper[2573]: I0709 16:00:34.980432 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c41bfea6-b89a-4572-9a27-8c517da2c352-maas-api-tls\") pod \"maas-api-e2e-mt-378f7a5c-79dd54ffd6-77rt8\" (UID: \"c41bfea6-b89a-4572-9a27-8c517da2c352\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-378f7a5c-79dd54ffd6-77rt8" Jul 09 16:00:34.989692 ip-10-0-133-183 kubenswrapper[2573]: I0709 16:00:34.989671 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hp7lt\" (UniqueName: \"kubernetes.io/projected/c41bfea6-b89a-4572-9a27-8c517da2c352-kube-api-access-hp7lt\") pod \"maas-api-e2e-mt-378f7a5c-79dd54ffd6-77rt8\" (UID: \"c41bfea6-b89a-4572-9a27-8c517da2c352\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-378f7a5c-79dd54ffd6-77rt8" Jul 09 16:00:35.115694 ip-10-0-133-183 kubenswrapper[2573]: I0709 16:00:35.115584 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-378f7a5c-79dd54ffd6-77rt8" Jul 09 16:00:35.254091 ip-10-0-133-183 kubenswrapper[2573]: I0709 16:00:35.254064 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-378f7a5c-79dd54ffd6-77rt8"] Jul 09 16:00:35.256659 ip-10-0-133-183 kubenswrapper[2573]: W0709 16:00:35.256630 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc41bfea6_b89a_4572_9a27_8c517da2c352.slice/crio-6fe241a408baeb1770789c7b76899c49444aa72d75edf430ae533c5518ef0796 WatchSource:0}: Error finding container 6fe241a408baeb1770789c7b76899c49444aa72d75edf430ae533c5518ef0796: Status 404 returned error can't find the container with id 6fe241a408baeb1770789c7b76899c49444aa72d75edf430ae533c5518ef0796 Jul 09 16:00:35.808427 ip-10-0-133-183 kubenswrapper[2573]: I0709 16:00:35.808385 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-378f7a5c-79dd54ffd6-77rt8" event={"ID":"c41bfea6-b89a-4572-9a27-8c517da2c352","Type":"ContainerStarted","Data":"c36309a5765492a363f9a3453d036f9a99f59fad78eaca8e589e14adea2e6b8b"} Jul 09 16:00:35.808996 ip-10-0-133-183 kubenswrapper[2573]: I0709 16:00:35.808964 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-378f7a5c-79dd54ffd6-77rt8" event={"ID":"c41bfea6-b89a-4572-9a27-8c517da2c352","Type":"ContainerStarted","Data":"6fe241a408baeb1770789c7b76899c49444aa72d75edf430ae533c5518ef0796"} Jul 09 16:00:35.809129 ip-10-0-133-183 kubenswrapper[2573]: I0709 16:00:35.809028 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-378f7a5c-79dd54ffd6-77rt8" Jul 09 16:00:35.810348 ip-10-0-133-183 kubenswrapper[2573]: I0709 16:00:35.810320 2573 generic.go:356] "Generic (PLEG): container finished" podID="d3bb0b8b-0bd1-4572-89aa-7c105fde52ab" containerID="0820c15392bd5df1d74f01227bae3d973d07ada2192aa41c797ca98ea7903c25" exitCode=137 Jul 09 16:00:35.810452 ip-10-0-133-183 kubenswrapper[2573]: I0709 16:00:35.810350 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-4d56b236-29726880-4wrdx" event={"ID":"d3bb0b8b-0bd1-4572-89aa-7c105fde52ab","Type":"ContainerDied","Data":"0820c15392bd5df1d74f01227bae3d973d07ada2192aa41c797ca98ea7903c25"} Jul 09 16:00:35.826153 ip-10-0-133-183 kubenswrapper[2573]: I0709 16:00:35.826086 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-mt-378f7a5c-79dd54ffd6-77rt8" podStartSLOduration=1.361124125 podStartE2EDuration="1.826068125s" totalImagesPullingTime="464.944ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 16:00:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 16:00:35.825669975 +0000 UTC m=+1757.547176836" watchObservedRunningTime="2026-07-09 16:00:35.826068125 +0000 UTC m=+1757.547575033" Jul 09 16:00:35.835988 ip-10-0-133-183 kubenswrapper[2573]: I0709 16:00:35.835965 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-4d56b236-29726880-4wrdx" Jul 09 16:00:36.816930 ip-10-0-133-183 kubenswrapper[2573]: I0709 16:00:36.816893 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-4d56b236-29726880-4wrdx" event={"ID":"d3bb0b8b-0bd1-4572-89aa-7c105fde52ab","Type":"ContainerDied","Data":"345e6748ebb9d9eba1125ba67911239c530fc903742feaec704809213c09eaba"} Jul 09 16:00:36.817380 ip-10-0-133-183 kubenswrapper[2573]: I0709 16:00:36.816960 2573 scope.go:117] "RemoveContainer" containerID="0820c15392bd5df1d74f01227bae3d973d07ada2192aa41c797ca98ea7903c25" Jul 09 16:00:36.817380 ip-10-0-133-183 kubenswrapper[2573]: I0709 16:00:36.816898 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-4d56b236-29726880-4wrdx" Jul 09 16:00:36.836084 ip-10-0-133-183 kubenswrapper[2573]: I0709 16:00:36.836056 2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-4d56b236-29726880-4wrdx"] Jul 09 16:00:36.842437 ip-10-0-133-183 kubenswrapper[2573]: I0709 16:00:36.842411 2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-4d56b236-29726880-4wrdx"] Jul 09 16:00:38.810622 ip-10-0-133-183 kubenswrapper[2573]: I0709 16:00:38.810586 2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d3bb0b8b-0bd1-4572-89aa-7c105fde52ab" path="/var/lib/kubelet/pods/d3bb0b8b-0bd1-4572-89aa-7c105fde52ab/volumes" Jul 09 16:00:42.828617 ip-10-0-133-183 kubenswrapper[2573]: I0709 16:00:42.828587 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-378f7a5c-79dd54ffd6-77rt8" Jul 09 16:01:00.011379 ip-10-0-133-183 kubenswrapper[2573]: I0709 16:01:00.011346 2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29726880-mwwxt"] Jul 09 16:01:00.011790 ip-10-0-133-183 kubenswrapper[2573]: I0709 16:01:00.011539 2573 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29726880-mwwxt" podUID="8cbe8ee2-5106-4a49-8035-cf751d58496d" containerName="cleanup" containerID="cri-o://57e1fb0023ec09bdd4fbf8fc58f62716515f71081387b41160425a8ffd226910" gracePeriod=30 Jul 09 16:01:18.800043 ip-10-0-133-183 kubenswrapper[2573]: I0709 16:01:18.799921 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-b7b9v_db22f88d-a953-488e-b384-67f991351f73/ovn-acl-logging/0.log" Jul 09 16:01:18.812353 ip-10-0-133-183 kubenswrapper[2573]: I0709 16:01:18.812323 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-b7b9v_db22f88d-a953-488e-b384-67f991351f73/ovn-acl-logging/0.log" Jul 09 16:01:22.312526 ip-10-0-133-183 kubenswrapper[2573]: I0709 16:01:22.312474 2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-378f7a5c-79dd54ffd6-77rt8"] Jul 09 16:01:22.313035 ip-10-0-133-183 kubenswrapper[2573]: I0709 16:01:22.312712 2573 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-mt-378f7a5c-79dd54ffd6-77rt8" podUID="c41bfea6-b89a-4572-9a27-8c517da2c352" containerName="maas-api" containerID="cri-o://c36309a5765492a363f9a3453d036f9a99f59fad78eaca8e589e14adea2e6b8b" gracePeriod=30 Jul 09 16:01:22.568374 ip-10-0-133-183 kubenswrapper[2573]: I0709 16:01:22.568311 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-378f7a5c-79dd54ffd6-77rt8" Jul 09 16:01:22.602727 ip-10-0-133-183 kubenswrapper[2573]: I0709 16:01:22.602695 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c41bfea6-b89a-4572-9a27-8c517da2c352-openshift-service-ca\") pod \"c41bfea6-b89a-4572-9a27-8c517da2c352\" (UID: \"c41bfea6-b89a-4572-9a27-8c517da2c352\") " Jul 09 16:01:22.602905 ip-10-0-133-183 kubenswrapper[2573]: I0709 16:01:22.602736 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hp7lt\" (UniqueName: \"kubernetes.io/projected/c41bfea6-b89a-4572-9a27-8c517da2c352-kube-api-access-hp7lt\") pod \"c41bfea6-b89a-4572-9a27-8c517da2c352\" (UID: \"c41bfea6-b89a-4572-9a27-8c517da2c352\") " Jul 09 16:01:22.602905 ip-10-0-133-183 kubenswrapper[2573]: I0709 16:01:22.602795 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c41bfea6-b89a-4572-9a27-8c517da2c352-maas-api-tls\") pod \"c41bfea6-b89a-4572-9a27-8c517da2c352\" (UID: \"c41bfea6-b89a-4572-9a27-8c517da2c352\") " Jul 09 16:01:22.603110 ip-10-0-133-183 kubenswrapper[2573]: I0709 16:01:22.603090 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c41bfea6-b89a-4572-9a27-8c517da2c352-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "c41bfea6-b89a-4572-9a27-8c517da2c352" (UID: "c41bfea6-b89a-4572-9a27-8c517da2c352"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 09 16:01:22.604979 ip-10-0-133-183 kubenswrapper[2573]: I0709 16:01:22.604954 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c41bfea6-b89a-4572-9a27-8c517da2c352-kube-api-access-hp7lt" (OuterVolumeSpecName: "kube-api-access-hp7lt") pod "c41bfea6-b89a-4572-9a27-8c517da2c352" (UID: "c41bfea6-b89a-4572-9a27-8c517da2c352"). InnerVolumeSpecName "kube-api-access-hp7lt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 09 16:01:22.604979 ip-10-0-133-183 kubenswrapper[2573]: I0709 16:01:22.604964 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c41bfea6-b89a-4572-9a27-8c517da2c352-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "c41bfea6-b89a-4572-9a27-8c517da2c352" (UID: "c41bfea6-b89a-4572-9a27-8c517da2c352"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 09 16:01:22.703405 ip-10-0-133-183 kubenswrapper[2573]: I0709 16:01:22.703370 2573 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c41bfea6-b89a-4572-9a27-8c517da2c352-openshift-service-ca\") on node \"ip-10-0-133-183.ec2.internal\" DevicePath \"\"" Jul 09 16:01:22.703405 ip-10-0-133-183 kubenswrapper[2573]: I0709 16:01:22.703398 2573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hp7lt\" (UniqueName: \"kubernetes.io/projected/c41bfea6-b89a-4572-9a27-8c517da2c352-kube-api-access-hp7lt\") on node \"ip-10-0-133-183.ec2.internal\" DevicePath \"\"" Jul 09 16:01:22.703405 ip-10-0-133-183 kubenswrapper[2573]: I0709 16:01:22.703409 2573 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c41bfea6-b89a-4572-9a27-8c517da2c352-maas-api-tls\") on node \"ip-10-0-133-183.ec2.internal\" DevicePath \"\"" Jul 09 16:01:23.044555 ip-10-0-133-183 kubenswrapper[2573]: I0709 16:01:23.044515 2573 generic.go:356] "Generic (PLEG): container finished" podID="c41bfea6-b89a-4572-9a27-8c517da2c352" containerID="c36309a5765492a363f9a3453d036f9a99f59fad78eaca8e589e14adea2e6b8b" exitCode=0 Jul 09 16:01:23.044728 ip-10-0-133-183 kubenswrapper[2573]: I0709 16:01:23.044563 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-378f7a5c-79dd54ffd6-77rt8" Jul 09 16:01:23.044728 ip-10-0-133-183 kubenswrapper[2573]: I0709 16:01:23.044596 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-378f7a5c-79dd54ffd6-77rt8" event={"ID":"c41bfea6-b89a-4572-9a27-8c517da2c352","Type":"ContainerDied","Data":"c36309a5765492a363f9a3453d036f9a99f59fad78eaca8e589e14adea2e6b8b"} Jul 09 16:01:23.044728 ip-10-0-133-183 kubenswrapper[2573]: I0709 16:01:23.044649 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-378f7a5c-79dd54ffd6-77rt8" event={"ID":"c41bfea6-b89a-4572-9a27-8c517da2c352","Type":"ContainerDied","Data":"6fe241a408baeb1770789c7b76899c49444aa72d75edf430ae533c5518ef0796"} Jul 09 16:01:23.044728 ip-10-0-133-183 kubenswrapper[2573]: I0709 16:01:23.044673 2573 scope.go:117] "RemoveContainer" containerID="c36309a5765492a363f9a3453d036f9a99f59fad78eaca8e589e14adea2e6b8b" Jul 09 16:01:23.053105 ip-10-0-133-183 kubenswrapper[2573]: I0709 16:01:23.053084 2573 scope.go:117] "RemoveContainer" containerID="c36309a5765492a363f9a3453d036f9a99f59fad78eaca8e589e14adea2e6b8b" Jul 09 16:01:23.053379 ip-10-0-133-183 kubenswrapper[2573]: E0709 16:01:23.053353 2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c36309a5765492a363f9a3453d036f9a99f59fad78eaca8e589e14adea2e6b8b\": container with ID starting with c36309a5765492a363f9a3453d036f9a99f59fad78eaca8e589e14adea2e6b8b not found: ID does not exist" containerID="c36309a5765492a363f9a3453d036f9a99f59fad78eaca8e589e14adea2e6b8b" Jul 09 16:01:23.053476 ip-10-0-133-183 kubenswrapper[2573]: I0709 16:01:23.053383 2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c36309a5765492a363f9a3453d036f9a99f59fad78eaca8e589e14adea2e6b8b"} err="failed to get container status \"c36309a5765492a363f9a3453d036f9a99f59fad78eaca8e589e14adea2e6b8b\": rpc error: code = NotFound desc = could not find container \"c36309a5765492a363f9a3453d036f9a99f59fad78eaca8e589e14adea2e6b8b\": container with ID starting with c36309a5765492a363f9a3453d036f9a99f59fad78eaca8e589e14adea2e6b8b not found: ID does not exist" Jul 09 16:01:23.067997 ip-10-0-133-183 kubenswrapper[2573]: I0709 16:01:23.065948 2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-378f7a5c-79dd54ffd6-77rt8"] Jul 09 16:01:23.070954 ip-10-0-133-183 kubenswrapper[2573]: I0709 16:01:23.070922 2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-378f7a5c-79dd54ffd6-77rt8"] Jul 09 16:01:24.810806 ip-10-0-133-183 kubenswrapper[2573]: I0709 16:01:24.810768 2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c41bfea6-b89a-4572-9a27-8c517da2c352" path="/var/lib/kubelet/pods/c41bfea6-b89a-4572-9a27-8c517da2c352/volumes" Jul 09 16:01:30.079546 ip-10-0-133-183 kubenswrapper[2573]: I0709 16:01:30.079507 2573 generic.go:356] "Generic (PLEG): container finished" podID="8cbe8ee2-5106-4a49-8035-cf751d58496d" containerID="57e1fb0023ec09bdd4fbf8fc58f62716515f71081387b41160425a8ffd226910" exitCode=137 Jul 09 16:01:30.079989 ip-10-0-133-183 kubenswrapper[2573]: I0709 16:01:30.079571 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29726880-mwwxt" event={"ID":"8cbe8ee2-5106-4a49-8035-cf751d58496d","Type":"ContainerDied","Data":"57e1fb0023ec09bdd4fbf8fc58f62716515f71081387b41160425a8ffd226910"} Jul 09 16:01:30.196763 ip-10-0-133-183 kubenswrapper[2573]: I0709 16:01:30.196734 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29726880-mwwxt" Jul 09 16:01:31.086021 ip-10-0-133-183 kubenswrapper[2573]: I0709 16:01:31.085919 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29726880-mwwxt" event={"ID":"8cbe8ee2-5106-4a49-8035-cf751d58496d","Type":"ContainerDied","Data":"dc3408af7729cce099410f36aafa567cb874858b24ac287529985ac7b0652387"} Jul 09 16:01:31.086021 ip-10-0-133-183 kubenswrapper[2573]: I0709 16:01:31.085958 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29726880-mwwxt" Jul 09 16:01:31.086021 ip-10-0-133-183 kubenswrapper[2573]: I0709 16:01:31.086018 2573 scope.go:117] "RemoveContainer" containerID="57e1fb0023ec09bdd4fbf8fc58f62716515f71081387b41160425a8ffd226910" Jul 09 16:01:31.105488 ip-10-0-133-183 kubenswrapper[2573]: I0709 16:01:31.105458 2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29726880-mwwxt"] Jul 09 16:01:31.111962 ip-10-0-133-183 kubenswrapper[2573]: I0709 16:01:31.111938 2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29726880-mwwxt"] Jul 09 16:01:32.810609 ip-10-0-133-183 kubenswrapper[2573]: I0709 16:01:32.810575 2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8cbe8ee2-5106-4a49-8035-cf751d58496d" path="/var/lib/kubelet/pods/8cbe8ee2-5106-4a49-8035-cf751d58496d/volumes" Jul 09 16:03:17.830285 ip-10-0-133-183 kubenswrapper[2573]: I0709 16:03:17.830244 2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-35b5e6-64b57c7d8-snkkr"] Jul 09 16:03:17.830995 ip-10-0-133-183 kubenswrapper[2573]: I0709 16:03:17.830604 2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8cbe8ee2-5106-4a49-8035-cf751d58496d" containerName="cleanup" Jul 09 16:03:17.830995 ip-10-0-133-183 kubenswrapper[2573]: I0709 16:03:17.830617 2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="8cbe8ee2-5106-4a49-8035-cf751d58496d" containerName="cleanup" Jul 09 16:03:17.830995 ip-10-0-133-183 kubenswrapper[2573]: I0709 16:03:17.830633 2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c41bfea6-b89a-4572-9a27-8c517da2c352" containerName="maas-api" Jul 09 16:03:17.830995 ip-10-0-133-183 kubenswrapper[2573]: I0709 16:03:17.830639 2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="c41bfea6-b89a-4572-9a27-8c517da2c352" containerName="maas-api" Jul 09 16:03:17.830995 ip-10-0-133-183 kubenswrapper[2573]: I0709 16:03:17.830647 2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d3bb0b8b-0bd1-4572-89aa-7c105fde52ab" containerName="cleanup" Jul 09 16:03:17.830995 ip-10-0-133-183 kubenswrapper[2573]: I0709 16:03:17.830652 2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="d3bb0b8b-0bd1-4572-89aa-7c105fde52ab" containerName="cleanup" Jul 09 16:03:17.830995 ip-10-0-133-183 kubenswrapper[2573]: I0709 16:03:17.830714 2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="8cbe8ee2-5106-4a49-8035-cf751d58496d" containerName="cleanup" Jul 09 16:03:17.830995 ip-10-0-133-183 kubenswrapper[2573]: I0709 16:03:17.830724 2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="c41bfea6-b89a-4572-9a27-8c517da2c352" containerName="maas-api" Jul 09 16:03:17.830995 ip-10-0-133-183 kubenswrapper[2573]: I0709 16:03:17.830731 2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="d3bb0b8b-0bd1-4572-89aa-7c105fde52ab" containerName="cleanup" Jul 09 16:03:17.832671 ip-10-0-133-183 kubenswrapper[2573]: I0709 16:03:17.832654 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-35b5e6-64b57c7d8-snkkr" Jul 09 16:03:17.835289 ip-10-0-133-183 kubenswrapper[2573]: I0709 16:03:17.835267 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-a-35b5e6-serving-cert\"" Jul 09 16:03:17.845280 ip-10-0-133-183 kubenswrapper[2573]: I0709 16:03:17.845249 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-35b5e6-64b57c7d8-snkkr"] Jul 09 16:03:17.954514 ip-10-0-133-183 kubenswrapper[2573]: I0709 16:03:17.954478 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/61e77f28-7084-4f5c-a7ce-9ce9a2c6b9cd-maas-api-tls\") pod \"maas-api-e2e-inf-a-35b5e6-64b57c7d8-snkkr\" (UID: \"61e77f28-7084-4f5c-a7ce-9ce9a2c6b9cd\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-35b5e6-64b57c7d8-snkkr" Jul 09 16:03:17.954514 ip-10-0-133-183 kubenswrapper[2573]: I0709 16:03:17.954513 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jpwmt\" (UniqueName: \"kubernetes.io/projected/61e77f28-7084-4f5c-a7ce-9ce9a2c6b9cd-kube-api-access-jpwmt\") pod \"maas-api-e2e-inf-a-35b5e6-64b57c7d8-snkkr\" (UID: \"61e77f28-7084-4f5c-a7ce-9ce9a2c6b9cd\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-35b5e6-64b57c7d8-snkkr" Jul 09 16:03:17.954721 ip-10-0-133-183 kubenswrapper[2573]: I0709 16:03:17.954635 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/61e77f28-7084-4f5c-a7ce-9ce9a2c6b9cd-openshift-service-ca\") pod \"maas-api-e2e-inf-a-35b5e6-64b57c7d8-snkkr\" (UID: \"61e77f28-7084-4f5c-a7ce-9ce9a2c6b9cd\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-35b5e6-64b57c7d8-snkkr" Jul 09 16:03:18.055300 ip-10-0-133-183 kubenswrapper[2573]: I0709 16:03:18.055264 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/61e77f28-7084-4f5c-a7ce-9ce9a2c6b9cd-maas-api-tls\") pod \"maas-api-e2e-inf-a-35b5e6-64b57c7d8-snkkr\" (UID: \"61e77f28-7084-4f5c-a7ce-9ce9a2c6b9cd\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-35b5e6-64b57c7d8-snkkr" Jul 09 16:03:18.055300 ip-10-0-133-183 kubenswrapper[2573]: I0709 16:03:18.055303 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jpwmt\" (UniqueName: \"kubernetes.io/projected/61e77f28-7084-4f5c-a7ce-9ce9a2c6b9cd-kube-api-access-jpwmt\") pod \"maas-api-e2e-inf-a-35b5e6-64b57c7d8-snkkr\" (UID: \"61e77f28-7084-4f5c-a7ce-9ce9a2c6b9cd\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-35b5e6-64b57c7d8-snkkr" Jul 09 16:03:18.055576 ip-10-0-133-183 kubenswrapper[2573]: I0709 16:03:18.055363 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/61e77f28-7084-4f5c-a7ce-9ce9a2c6b9cd-openshift-service-ca\") pod \"maas-api-e2e-inf-a-35b5e6-64b57c7d8-snkkr\" (UID: \"61e77f28-7084-4f5c-a7ce-9ce9a2c6b9cd\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-35b5e6-64b57c7d8-snkkr" Jul 09 16:03:18.055576 ip-10-0-133-183 kubenswrapper[2573]: E0709 16:03:18.055446 2573 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-inf-a-35b5e6-serving-cert: secret "maas-api-e2e-inf-a-35b5e6-serving-cert" not found Jul 09 16:03:18.055576 ip-10-0-133-183 kubenswrapper[2573]: E0709 16:03:18.055539 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/61e77f28-7084-4f5c-a7ce-9ce9a2c6b9cd-maas-api-tls podName:61e77f28-7084-4f5c-a7ce-9ce9a2c6b9cd nodeName:}" failed. No retries permitted until 2026-07-09 16:03:18.555515298 +0000 UTC m=+1920.277022145 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/61e77f28-7084-4f5c-a7ce-9ce9a2c6b9cd-maas-api-tls") pod "maas-api-e2e-inf-a-35b5e6-64b57c7d8-snkkr" (UID: "61e77f28-7084-4f5c-a7ce-9ce9a2c6b9cd") : secret "maas-api-e2e-inf-a-35b5e6-serving-cert" not found Jul 09 16:03:18.056064 ip-10-0-133-183 kubenswrapper[2573]: I0709 16:03:18.056047 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/61e77f28-7084-4f5c-a7ce-9ce9a2c6b9cd-openshift-service-ca\") pod \"maas-api-e2e-inf-a-35b5e6-64b57c7d8-snkkr\" (UID: \"61e77f28-7084-4f5c-a7ce-9ce9a2c6b9cd\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-35b5e6-64b57c7d8-snkkr" Jul 09 16:03:18.068323 ip-10-0-133-183 kubenswrapper[2573]: I0709 16:03:18.068294 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jpwmt\" (UniqueName: \"kubernetes.io/projected/61e77f28-7084-4f5c-a7ce-9ce9a2c6b9cd-kube-api-access-jpwmt\") pod \"maas-api-e2e-inf-a-35b5e6-64b57c7d8-snkkr\" (UID: \"61e77f28-7084-4f5c-a7ce-9ce9a2c6b9cd\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-35b5e6-64b57c7d8-snkkr" Jul 09 16:03:18.560336 ip-10-0-133-183 kubenswrapper[2573]: I0709 16:03:18.560300 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/61e77f28-7084-4f5c-a7ce-9ce9a2c6b9cd-maas-api-tls\") pod \"maas-api-e2e-inf-a-35b5e6-64b57c7d8-snkkr\" (UID: \"61e77f28-7084-4f5c-a7ce-9ce9a2c6b9cd\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-35b5e6-64b57c7d8-snkkr" Jul 09 16:03:18.562927 ip-10-0-133-183 kubenswrapper[2573]: I0709 16:03:18.562898 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/61e77f28-7084-4f5c-a7ce-9ce9a2c6b9cd-maas-api-tls\") pod \"maas-api-e2e-inf-a-35b5e6-64b57c7d8-snkkr\" (UID: \"61e77f28-7084-4f5c-a7ce-9ce9a2c6b9cd\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-35b5e6-64b57c7d8-snkkr" Jul 09 16:03:18.744590 ip-10-0-133-183 kubenswrapper[2573]: I0709 16:03:18.744551 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-35b5e6-64b57c7d8-snkkr" Jul 09 16:03:18.889811 ip-10-0-133-183 kubenswrapper[2573]: I0709 16:03:18.889785 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-35b5e6-64b57c7d8-snkkr"] Jul 09 16:03:18.892420 ip-10-0-133-183 kubenswrapper[2573]: W0709 16:03:18.892394 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod61e77f28_7084_4f5c_a7ce_9ce9a2c6b9cd.slice/crio-e9db2e1a1fe791598692f882f2035ea559d7da00612f84851a471117174dd56c WatchSource:0}: Error finding container e9db2e1a1fe791598692f882f2035ea559d7da00612f84851a471117174dd56c: Status 404 returned error can't find the container with id e9db2e1a1fe791598692f882f2035ea559d7da00612f84851a471117174dd56c Jul 09 16:03:18.894106 ip-10-0-133-183 kubenswrapper[2573]: I0709 16:03:18.894088 2573 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 09 16:03:19.590652 ip-10-0-133-183 kubenswrapper[2573]: I0709 16:03:19.590618 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-35b5e6-64b57c7d8-snkkr" event={"ID":"61e77f28-7084-4f5c-a7ce-9ce9a2c6b9cd","Type":"ContainerStarted","Data":"e9db2e1a1fe791598692f882f2035ea559d7da00612f84851a471117174dd56c"} Jul 09 16:03:20.596635 ip-10-0-133-183 kubenswrapper[2573]: I0709 16:03:20.596589 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-35b5e6-64b57c7d8-snkkr" event={"ID":"61e77f28-7084-4f5c-a7ce-9ce9a2c6b9cd","Type":"ContainerStarted","Data":"6ec67fd7aad08aa5ad5159392ab0e6dacdb8fc4dcd70952ff35136ec58b4929a"} Jul 09 16:03:20.597053 ip-10-0-133-183 kubenswrapper[2573]: I0709 16:03:20.596662 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-35b5e6-64b57c7d8-snkkr" Jul 09 16:03:20.615510 ip-10-0-133-183 kubenswrapper[2573]: I0709 16:03:20.615464 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-35b5e6-64b57c7d8-snkkr" podStartSLOduration=2.886800243 podStartE2EDuration="3.615446796s" totalImagesPullingTime="728.646553ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 16:03:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 16:03:20.613926358 +0000 UTC m=+1922.335433244" watchObservedRunningTime="2026-07-09 16:03:20.615446796 +0000 UTC m=+1922.336953663" Jul 09 16:03:25.802742 ip-10-0-133-183 kubenswrapper[2573]: I0709 16:03:25.802708 2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-2ca941-6b6c8c6985-wblhq"] Jul 09 16:03:25.806413 ip-10-0-133-183 kubenswrapper[2573]: I0709 16:03:25.806389 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-2ca941-6b6c8c6985-wblhq" Jul 09 16:03:25.809123 ip-10-0-133-183 kubenswrapper[2573]: I0709 16:03:25.809103 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-b-2ca941-serving-cert\"" Jul 09 16:03:25.817223 ip-10-0-133-183 kubenswrapper[2573]: I0709 16:03:25.817200 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-2ca941-6b6c8c6985-wblhq"] Jul 09 16:03:25.929965 ip-10-0-133-183 kubenswrapper[2573]: I0709 16:03:25.929928 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6b436de9-5c37-41e1-a2ba-62b851324463-maas-api-tls\") pod \"maas-api-e2e-inf-b-2ca941-6b6c8c6985-wblhq\" (UID: \"6b436de9-5c37-41e1-a2ba-62b851324463\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-2ca941-6b6c8c6985-wblhq" Jul 09 16:03:25.930139 ip-10-0-133-183 kubenswrapper[2573]: I0709 16:03:25.930054 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6b436de9-5c37-41e1-a2ba-62b851324463-openshift-service-ca\") pod \"maas-api-e2e-inf-b-2ca941-6b6c8c6985-wblhq\" (UID: \"6b436de9-5c37-41e1-a2ba-62b851324463\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-2ca941-6b6c8c6985-wblhq" Jul 09 16:03:25.930139 ip-10-0-133-183 kubenswrapper[2573]: I0709 16:03:25.930086 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g9sls\" (UniqueName: \"kubernetes.io/projected/6b436de9-5c37-41e1-a2ba-62b851324463-kube-api-access-g9sls\") pod \"maas-api-e2e-inf-b-2ca941-6b6c8c6985-wblhq\" (UID: \"6b436de9-5c37-41e1-a2ba-62b851324463\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-2ca941-6b6c8c6985-wblhq" Jul 09 16:03:26.031234 ip-10-0-133-183 kubenswrapper[2573]: I0709 16:03:26.031191 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6b436de9-5c37-41e1-a2ba-62b851324463-openshift-service-ca\") pod \"maas-api-e2e-inf-b-2ca941-6b6c8c6985-wblhq\" (UID: \"6b436de9-5c37-41e1-a2ba-62b851324463\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-2ca941-6b6c8c6985-wblhq" Jul 09 16:03:26.031234 ip-10-0-133-183 kubenswrapper[2573]: I0709 16:03:26.031234 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g9sls\" (UniqueName: \"kubernetes.io/projected/6b436de9-5c37-41e1-a2ba-62b851324463-kube-api-access-g9sls\") pod \"maas-api-e2e-inf-b-2ca941-6b6c8c6985-wblhq\" (UID: \"6b436de9-5c37-41e1-a2ba-62b851324463\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-2ca941-6b6c8c6985-wblhq" Jul 09 16:03:26.031447 ip-10-0-133-183 kubenswrapper[2573]: I0709 16:03:26.031286 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6b436de9-5c37-41e1-a2ba-62b851324463-maas-api-tls\") pod \"maas-api-e2e-inf-b-2ca941-6b6c8c6985-wblhq\" (UID: \"6b436de9-5c37-41e1-a2ba-62b851324463\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-2ca941-6b6c8c6985-wblhq" Jul 09 16:03:26.032009 ip-10-0-133-183 kubenswrapper[2573]: I0709 16:03:26.031989 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6b436de9-5c37-41e1-a2ba-62b851324463-openshift-service-ca\") pod \"maas-api-e2e-inf-b-2ca941-6b6c8c6985-wblhq\" (UID: \"6b436de9-5c37-41e1-a2ba-62b851324463\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-2ca941-6b6c8c6985-wblhq" Jul 09 16:03:26.033871 ip-10-0-133-183 kubenswrapper[2573]: I0709 16:03:26.033851 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6b436de9-5c37-41e1-a2ba-62b851324463-maas-api-tls\") pod \"maas-api-e2e-inf-b-2ca941-6b6c8c6985-wblhq\" (UID: \"6b436de9-5c37-41e1-a2ba-62b851324463\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-2ca941-6b6c8c6985-wblhq" Jul 09 16:03:26.043010 ip-10-0-133-183 kubenswrapper[2573]: I0709 16:03:26.042983 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-g9sls\" (UniqueName: \"kubernetes.io/projected/6b436de9-5c37-41e1-a2ba-62b851324463-kube-api-access-g9sls\") pod \"maas-api-e2e-inf-b-2ca941-6b6c8c6985-wblhq\" (UID: \"6b436de9-5c37-41e1-a2ba-62b851324463\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-2ca941-6b6c8c6985-wblhq" Jul 09 16:03:26.118789 ip-10-0-133-183 kubenswrapper[2573]: I0709 16:03:26.118702 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-2ca941-6b6c8c6985-wblhq" Jul 09 16:03:26.259746 ip-10-0-133-183 kubenswrapper[2573]: I0709 16:03:26.259721 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-2ca941-6b6c8c6985-wblhq"] Jul 09 16:03:26.262232 ip-10-0-133-183 kubenswrapper[2573]: W0709 16:03:26.262192 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6b436de9_5c37_41e1_a2ba_62b851324463.slice/crio-3cd8cd7df94975a877da0f685ef6aec7984635abda4fcd690fd858cd2c49f575 WatchSource:0}: Error finding container 3cd8cd7df94975a877da0f685ef6aec7984635abda4fcd690fd858cd2c49f575: Status 404 returned error can't find the container with id 3cd8cd7df94975a877da0f685ef6aec7984635abda4fcd690fd858cd2c49f575 Jul 09 16:03:26.606719 ip-10-0-133-183 kubenswrapper[2573]: I0709 16:03:26.606691 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-35b5e6-64b57c7d8-snkkr" Jul 09 16:03:26.625732 ip-10-0-133-183 kubenswrapper[2573]: I0709 16:03:26.625701 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-2ca941-6b6c8c6985-wblhq" event={"ID":"6b436de9-5c37-41e1-a2ba-62b851324463","Type":"ContainerStarted","Data":"3cd8cd7df94975a877da0f685ef6aec7984635abda4fcd690fd858cd2c49f575"} Jul 09 16:03:27.631308 ip-10-0-133-183 kubenswrapper[2573]: I0709 16:03:27.631273 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-2ca941-6b6c8c6985-wblhq" event={"ID":"6b436de9-5c37-41e1-a2ba-62b851324463","Type":"ContainerStarted","Data":"2901d4d093ac470186eb913a917087a111816b8eedd631d19f3fa23fd2224b72"} Jul 09 16:03:27.631777 ip-10-0-133-183 kubenswrapper[2573]: I0709 16:03:27.631408 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-2ca941-6b6c8c6985-wblhq" Jul 09 16:03:27.649528 ip-10-0-133-183 kubenswrapper[2573]: I0709 16:03:27.649473 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-2ca941-6b6c8c6985-wblhq" podStartSLOduration=2.205185998 podStartE2EDuration="2.649456401s" totalImagesPullingTime="444.270403ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 16:03:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 16:03:27.64936035 +0000 UTC m=+1929.370867250" watchObservedRunningTime="2026-07-09 16:03:27.649456401 +0000 UTC m=+1929.370963267" Jul 09 16:03:33.645852 ip-10-0-133-183 kubenswrapper[2573]: I0709 16:03:33.645820 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-2ca941-6b6c8c6985-wblhq" Jul 09 16:04:46.356319 ip-10-0-133-183 kubenswrapper[2573]: I0709 16:04:46.356281 2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-35b5e6-64b57c7d8-snkkr"] Jul 09 16:04:46.356888 ip-10-0-133-183 kubenswrapper[2573]: I0709 16:04:46.356530 2573 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-35b5e6-64b57c7d8-snkkr" podUID="61e77f28-7084-4f5c-a7ce-9ce9a2c6b9cd" containerName="maas-api" containerID="cri-o://6ec67fd7aad08aa5ad5159392ab0e6dacdb8fc4dcd70952ff35136ec58b4929a" gracePeriod=30 Jul 09 16:04:46.605296 ip-10-0-133-183 kubenswrapper[2573]: I0709 16:04:46.605260 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-35b5e6-64b57c7d8-snkkr" Jul 09 16:04:46.679516 ip-10-0-133-183 kubenswrapper[2573]: I0709 16:04:46.679415 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jpwmt\" (UniqueName: \"kubernetes.io/projected/61e77f28-7084-4f5c-a7ce-9ce9a2c6b9cd-kube-api-access-jpwmt\") pod \"61e77f28-7084-4f5c-a7ce-9ce9a2c6b9cd\" (UID: \"61e77f28-7084-4f5c-a7ce-9ce9a2c6b9cd\") " Jul 09 16:04:46.679516 ip-10-0-133-183 kubenswrapper[2573]: I0709 16:04:46.679518 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/61e77f28-7084-4f5c-a7ce-9ce9a2c6b9cd-openshift-service-ca\") pod \"61e77f28-7084-4f5c-a7ce-9ce9a2c6b9cd\" (UID: \"61e77f28-7084-4f5c-a7ce-9ce9a2c6b9cd\") " Jul 09 16:04:46.679734 ip-10-0-133-183 kubenswrapper[2573]: I0709 16:04:46.679541 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/61e77f28-7084-4f5c-a7ce-9ce9a2c6b9cd-maas-api-tls\") pod \"61e77f28-7084-4f5c-a7ce-9ce9a2c6b9cd\" (UID: \"61e77f28-7084-4f5c-a7ce-9ce9a2c6b9cd\") " Jul 09 16:04:46.679909 ip-10-0-133-183 kubenswrapper[2573]: I0709 16:04:46.679858 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/61e77f28-7084-4f5c-a7ce-9ce9a2c6b9cd-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "61e77f28-7084-4f5c-a7ce-9ce9a2c6b9cd" (UID: "61e77f28-7084-4f5c-a7ce-9ce9a2c6b9cd"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 09 16:04:46.681600 ip-10-0-133-183 kubenswrapper[2573]: I0709 16:04:46.681579 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/61e77f28-7084-4f5c-a7ce-9ce9a2c6b9cd-kube-api-access-jpwmt" (OuterVolumeSpecName: "kube-api-access-jpwmt") pod "61e77f28-7084-4f5c-a7ce-9ce9a2c6b9cd" (UID: "61e77f28-7084-4f5c-a7ce-9ce9a2c6b9cd"). InnerVolumeSpecName "kube-api-access-jpwmt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 09 16:04:46.681656 ip-10-0-133-183 kubenswrapper[2573]: I0709 16:04:46.681616 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/61e77f28-7084-4f5c-a7ce-9ce9a2c6b9cd-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "61e77f28-7084-4f5c-a7ce-9ce9a2c6b9cd" (UID: "61e77f28-7084-4f5c-a7ce-9ce9a2c6b9cd"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 09 16:04:46.781077 ip-10-0-133-183 kubenswrapper[2573]: I0709 16:04:46.781025 2573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jpwmt\" (UniqueName: \"kubernetes.io/projected/61e77f28-7084-4f5c-a7ce-9ce9a2c6b9cd-kube-api-access-jpwmt\") on node \"ip-10-0-133-183.ec2.internal\" DevicePath \"\"" Jul 09 16:04:46.781077 ip-10-0-133-183 kubenswrapper[2573]: I0709 16:04:46.781072 2573 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/61e77f28-7084-4f5c-a7ce-9ce9a2c6b9cd-openshift-service-ca\") on node \"ip-10-0-133-183.ec2.internal\" DevicePath \"\"" Jul 09 16:04:46.781077 ip-10-0-133-183 kubenswrapper[2573]: I0709 16:04:46.781085 2573 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/61e77f28-7084-4f5c-a7ce-9ce9a2c6b9cd-maas-api-tls\") on node \"ip-10-0-133-183.ec2.internal\" DevicePath \"\"" Jul 09 16:04:47.017432 ip-10-0-133-183 kubenswrapper[2573]: I0709 16:04:47.017398 2573 generic.go:356] "Generic (PLEG): container finished" podID="61e77f28-7084-4f5c-a7ce-9ce9a2c6b9cd" containerID="6ec67fd7aad08aa5ad5159392ab0e6dacdb8fc4dcd70952ff35136ec58b4929a" exitCode=0 Jul 09 16:04:47.017432 ip-10-0-133-183 kubenswrapper[2573]: I0709 16:04:47.017436 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-35b5e6-64b57c7d8-snkkr" event={"ID":"61e77f28-7084-4f5c-a7ce-9ce9a2c6b9cd","Type":"ContainerDied","Data":"6ec67fd7aad08aa5ad5159392ab0e6dacdb8fc4dcd70952ff35136ec58b4929a"} Jul 09 16:04:47.017689 ip-10-0-133-183 kubenswrapper[2573]: I0709 16:04:47.017449 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-35b5e6-64b57c7d8-snkkr" Jul 09 16:04:47.017689 ip-10-0-133-183 kubenswrapper[2573]: I0709 16:04:47.017486 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-35b5e6-64b57c7d8-snkkr" event={"ID":"61e77f28-7084-4f5c-a7ce-9ce9a2c6b9cd","Type":"ContainerDied","Data":"e9db2e1a1fe791598692f882f2035ea559d7da00612f84851a471117174dd56c"} Jul 09 16:04:47.017689 ip-10-0-133-183 kubenswrapper[2573]: I0709 16:04:47.017515 2573 scope.go:117] "RemoveContainer" containerID="6ec67fd7aad08aa5ad5159392ab0e6dacdb8fc4dcd70952ff35136ec58b4929a" Jul 09 16:04:47.025808 ip-10-0-133-183 kubenswrapper[2573]: I0709 16:04:47.025788 2573 scope.go:117] "RemoveContainer" containerID="6ec67fd7aad08aa5ad5159392ab0e6dacdb8fc4dcd70952ff35136ec58b4929a" Jul 09 16:04:47.026074 ip-10-0-133-183 kubenswrapper[2573]: E0709 16:04:47.026055 2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6ec67fd7aad08aa5ad5159392ab0e6dacdb8fc4dcd70952ff35136ec58b4929a\": container with ID starting with 6ec67fd7aad08aa5ad5159392ab0e6dacdb8fc4dcd70952ff35136ec58b4929a not found: ID does not exist" containerID="6ec67fd7aad08aa5ad5159392ab0e6dacdb8fc4dcd70952ff35136ec58b4929a" Jul 09 16:04:47.026123 ip-10-0-133-183 kubenswrapper[2573]: I0709 16:04:47.026082 2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6ec67fd7aad08aa5ad5159392ab0e6dacdb8fc4dcd70952ff35136ec58b4929a"} err="failed to get container status \"6ec67fd7aad08aa5ad5159392ab0e6dacdb8fc4dcd70952ff35136ec58b4929a\": rpc error: code = NotFound desc = could not find container \"6ec67fd7aad08aa5ad5159392ab0e6dacdb8fc4dcd70952ff35136ec58b4929a\": container with ID starting with 6ec67fd7aad08aa5ad5159392ab0e6dacdb8fc4dcd70952ff35136ec58b4929a not found: ID does not exist" Jul 09 16:04:47.038458 ip-10-0-133-183 kubenswrapper[2573]: I0709 16:04:47.038432 2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-35b5e6-64b57c7d8-snkkr"] Jul 09 16:04:47.048374 ip-10-0-133-183 kubenswrapper[2573]: I0709 16:04:47.048351 2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-35b5e6-64b57c7d8-snkkr"] Jul 09 16:04:48.817511 ip-10-0-133-183 kubenswrapper[2573]: I0709 16:04:48.817473 2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="61e77f28-7084-4f5c-a7ce-9ce9a2c6b9cd" path="/var/lib/kubelet/pods/61e77f28-7084-4f5c-a7ce-9ce9a2c6b9cd/volumes" Jul 09 16:05:30.565041 ip-10-0-133-183 kubenswrapper[2573]: I0709 16:05:30.564999 2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-2ca941-6b6c8c6985-wblhq"] Jul 09 16:05:30.565479 ip-10-0-133-183 kubenswrapper[2573]: I0709 16:05:30.565358 2573 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-2ca941-6b6c8c6985-wblhq" podUID="6b436de9-5c37-41e1-a2ba-62b851324463" containerName="maas-api" containerID="cri-o://2901d4d093ac470186eb913a917087a111816b8eedd631d19f3fa23fd2224b72" gracePeriod=30 Jul 09 16:05:30.807457 ip-10-0-133-183 kubenswrapper[2573]: I0709 16:05:30.807433 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-2ca941-6b6c8c6985-wblhq" Jul 09 16:05:30.882098 ip-10-0-133-183 kubenswrapper[2573]: I0709 16:05:30.882011 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6b436de9-5c37-41e1-a2ba-62b851324463-openshift-service-ca\") pod \"6b436de9-5c37-41e1-a2ba-62b851324463\" (UID: \"6b436de9-5c37-41e1-a2ba-62b851324463\") " Jul 09 16:05:30.882098 ip-10-0-133-183 kubenswrapper[2573]: I0709 16:05:30.882070 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-g9sls\" (UniqueName: \"kubernetes.io/projected/6b436de9-5c37-41e1-a2ba-62b851324463-kube-api-access-g9sls\") pod \"6b436de9-5c37-41e1-a2ba-62b851324463\" (UID: \"6b436de9-5c37-41e1-a2ba-62b851324463\") " Jul 09 16:05:30.882319 ip-10-0-133-183 kubenswrapper[2573]: I0709 16:05:30.882109 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6b436de9-5c37-41e1-a2ba-62b851324463-maas-api-tls\") pod \"6b436de9-5c37-41e1-a2ba-62b851324463\" (UID: \"6b436de9-5c37-41e1-a2ba-62b851324463\") " Jul 09 16:05:30.882396 ip-10-0-133-183 kubenswrapper[2573]: I0709 16:05:30.882373 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6b436de9-5c37-41e1-a2ba-62b851324463-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "6b436de9-5c37-41e1-a2ba-62b851324463" (UID: "6b436de9-5c37-41e1-a2ba-62b851324463"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 09 16:05:30.884216 ip-10-0-133-183 kubenswrapper[2573]: I0709 16:05:30.884189 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6b436de9-5c37-41e1-a2ba-62b851324463-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "6b436de9-5c37-41e1-a2ba-62b851324463" (UID: "6b436de9-5c37-41e1-a2ba-62b851324463"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 09 16:05:30.884314 ip-10-0-133-183 kubenswrapper[2573]: I0709 16:05:30.884194 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6b436de9-5c37-41e1-a2ba-62b851324463-kube-api-access-g9sls" (OuterVolumeSpecName: "kube-api-access-g9sls") pod "6b436de9-5c37-41e1-a2ba-62b851324463" (UID: "6b436de9-5c37-41e1-a2ba-62b851324463"). InnerVolumeSpecName "kube-api-access-g9sls". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 09 16:05:30.983625 ip-10-0-133-183 kubenswrapper[2573]: I0709 16:05:30.983569 2573 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6b436de9-5c37-41e1-a2ba-62b851324463-openshift-service-ca\") on node \"ip-10-0-133-183.ec2.internal\" DevicePath \"\"" Jul 09 16:05:30.983625 ip-10-0-133-183 kubenswrapper[2573]: I0709 16:05:30.983617 2573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g9sls\" (UniqueName: \"kubernetes.io/projected/6b436de9-5c37-41e1-a2ba-62b851324463-kube-api-access-g9sls\") on node \"ip-10-0-133-183.ec2.internal\" DevicePath \"\"" Jul 09 16:05:30.983625 ip-10-0-133-183 kubenswrapper[2573]: I0709 16:05:30.983628 2573 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6b436de9-5c37-41e1-a2ba-62b851324463-maas-api-tls\") on node \"ip-10-0-133-183.ec2.internal\" DevicePath \"\"" Jul 09 16:05:31.225441 ip-10-0-133-183 kubenswrapper[2573]: I0709 16:05:31.225406 2573 generic.go:356] "Generic (PLEG): container finished" podID="6b436de9-5c37-41e1-a2ba-62b851324463" containerID="2901d4d093ac470186eb913a917087a111816b8eedd631d19f3fa23fd2224b72" exitCode=0 Jul 09 16:05:31.225640 ip-10-0-133-183 kubenswrapper[2573]: I0709 16:05:31.225459 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-2ca941-6b6c8c6985-wblhq" Jul 09 16:05:31.225640 ip-10-0-133-183 kubenswrapper[2573]: I0709 16:05:31.225473 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-2ca941-6b6c8c6985-wblhq" event={"ID":"6b436de9-5c37-41e1-a2ba-62b851324463","Type":"ContainerDied","Data":"2901d4d093ac470186eb913a917087a111816b8eedd631d19f3fa23fd2224b72"} Jul 09 16:05:31.225640 ip-10-0-133-183 kubenswrapper[2573]: I0709 16:05:31.225515 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-2ca941-6b6c8c6985-wblhq" event={"ID":"6b436de9-5c37-41e1-a2ba-62b851324463","Type":"ContainerDied","Data":"3cd8cd7df94975a877da0f685ef6aec7984635abda4fcd690fd858cd2c49f575"} Jul 09 16:05:31.225640 ip-10-0-133-183 kubenswrapper[2573]: I0709 16:05:31.225539 2573 scope.go:117] "RemoveContainer" containerID="2901d4d093ac470186eb913a917087a111816b8eedd631d19f3fa23fd2224b72" Jul 09 16:05:31.235083 ip-10-0-133-183 kubenswrapper[2573]: I0709 16:05:31.235062 2573 scope.go:117] "RemoveContainer" containerID="2901d4d093ac470186eb913a917087a111816b8eedd631d19f3fa23fd2224b72" Jul 09 16:05:31.235354 ip-10-0-133-183 kubenswrapper[2573]: E0709 16:05:31.235334 2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2901d4d093ac470186eb913a917087a111816b8eedd631d19f3fa23fd2224b72\": container with ID starting with 2901d4d093ac470186eb913a917087a111816b8eedd631d19f3fa23fd2224b72 not found: ID does not exist" containerID="2901d4d093ac470186eb913a917087a111816b8eedd631d19f3fa23fd2224b72" Jul 09 16:05:31.235406 ip-10-0-133-183 kubenswrapper[2573]: I0709 16:05:31.235364 2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2901d4d093ac470186eb913a917087a111816b8eedd631d19f3fa23fd2224b72"} err="failed to get container status \"2901d4d093ac470186eb913a917087a111816b8eedd631d19f3fa23fd2224b72\": rpc error: code = NotFound desc = could not find container \"2901d4d093ac470186eb913a917087a111816b8eedd631d19f3fa23fd2224b72\": container with ID starting with 2901d4d093ac470186eb913a917087a111816b8eedd631d19f3fa23fd2224b72 not found: ID does not exist" Jul 09 16:05:31.251074 ip-10-0-133-183 kubenswrapper[2573]: I0709 16:05:31.251045 2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-2ca941-6b6c8c6985-wblhq"] Jul 09 16:05:31.255752 ip-10-0-133-183 kubenswrapper[2573]: I0709 16:05:31.255724 2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-2ca941-6b6c8c6985-wblhq"] Jul 09 16:05:32.811111 ip-10-0-133-183 kubenswrapper[2573]: I0709 16:05:32.811080 2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6b436de9-5c37-41e1-a2ba-62b851324463" path="/var/lib/kubelet/pods/6b436de9-5c37-41e1-a2ba-62b851324463/volumes" Jul 09 16:05:41.616499 ip-10-0-133-183 kubenswrapper[2573]: I0709 16:05:41.616418 2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-5363ce-9c8454d5b-rxrfp"] Jul 09 16:05:41.616904 ip-10-0-133-183 kubenswrapper[2573]: I0709 16:05:41.616887 2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="61e77f28-7084-4f5c-a7ce-9ce9a2c6b9cd" containerName="maas-api" Jul 09 16:05:41.616966 ip-10-0-133-183 kubenswrapper[2573]: I0709 16:05:41.616906 2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="61e77f28-7084-4f5c-a7ce-9ce9a2c6b9cd" containerName="maas-api" Jul 09 16:05:41.616966 ip-10-0-133-183 kubenswrapper[2573]: I0709 16:05:41.616916 2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6b436de9-5c37-41e1-a2ba-62b851324463" containerName="maas-api" Jul 09 16:05:41.616966 ip-10-0-133-183 kubenswrapper[2573]: I0709 16:05:41.616922 2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="6b436de9-5c37-41e1-a2ba-62b851324463" containerName="maas-api" Jul 09 16:05:41.617058 ip-10-0-133-183 kubenswrapper[2573]: I0709 16:05:41.616981 2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="6b436de9-5c37-41e1-a2ba-62b851324463" containerName="maas-api" Jul 09 16:05:41.617058 ip-10-0-133-183 kubenswrapper[2573]: I0709 16:05:41.616991 2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="61e77f28-7084-4f5c-a7ce-9ce9a2c6b9cd" containerName="maas-api" Jul 09 16:05:41.622472 ip-10-0-133-183 kubenswrapper[2573]: I0709 16:05:41.622439 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-5363ce-9c8454d5b-rxrfp" Jul 09 16:05:41.625319 ip-10-0-133-183 kubenswrapper[2573]: I0709 16:05:41.625291 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-a-5363ce-serving-cert\"" Jul 09 16:05:41.630216 ip-10-0-133-183 kubenswrapper[2573]: I0709 16:05:41.630193 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-5363ce-9c8454d5b-rxrfp"] Jul 09 16:05:41.688471 ip-10-0-133-183 kubenswrapper[2573]: I0709 16:05:41.688433 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/563a45cf-de38-4d55-875f-5079037e8e6c-maas-api-tls\") pod \"maas-api-e2e-api-a-5363ce-9c8454d5b-rxrfp\" (UID: \"563a45cf-de38-4d55-875f-5079037e8e6c\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-5363ce-9c8454d5b-rxrfp" Jul 09 16:05:41.688670 ip-10-0-133-183 kubenswrapper[2573]: I0709 16:05:41.688549 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/563a45cf-de38-4d55-875f-5079037e8e6c-openshift-service-ca\") pod \"maas-api-e2e-api-a-5363ce-9c8454d5b-rxrfp\" (UID: \"563a45cf-de38-4d55-875f-5079037e8e6c\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-5363ce-9c8454d5b-rxrfp" Jul 09 16:05:41.688670 ip-10-0-133-183 kubenswrapper[2573]: I0709 16:05:41.688589 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ffgp8\" (UniqueName: \"kubernetes.io/projected/563a45cf-de38-4d55-875f-5079037e8e6c-kube-api-access-ffgp8\") pod \"maas-api-e2e-api-a-5363ce-9c8454d5b-rxrfp\" (UID: \"563a45cf-de38-4d55-875f-5079037e8e6c\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-5363ce-9c8454d5b-rxrfp" Jul 09 16:05:41.789382 ip-10-0-133-183 kubenswrapper[2573]: I0709 16:05:41.789349 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/563a45cf-de38-4d55-875f-5079037e8e6c-openshift-service-ca\") pod \"maas-api-e2e-api-a-5363ce-9c8454d5b-rxrfp\" (UID: \"563a45cf-de38-4d55-875f-5079037e8e6c\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-5363ce-9c8454d5b-rxrfp" Jul 09 16:05:41.789382 ip-10-0-133-183 kubenswrapper[2573]: I0709 16:05:41.789397 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ffgp8\" (UniqueName: \"kubernetes.io/projected/563a45cf-de38-4d55-875f-5079037e8e6c-kube-api-access-ffgp8\") pod \"maas-api-e2e-api-a-5363ce-9c8454d5b-rxrfp\" (UID: \"563a45cf-de38-4d55-875f-5079037e8e6c\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-5363ce-9c8454d5b-rxrfp" Jul 09 16:05:41.789598 ip-10-0-133-183 kubenswrapper[2573]: I0709 16:05:41.789432 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/563a45cf-de38-4d55-875f-5079037e8e6c-maas-api-tls\") pod \"maas-api-e2e-api-a-5363ce-9c8454d5b-rxrfp\" (UID: \"563a45cf-de38-4d55-875f-5079037e8e6c\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-5363ce-9c8454d5b-rxrfp" Jul 09 16:05:41.790122 ip-10-0-133-183 kubenswrapper[2573]: I0709 16:05:41.790085 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/563a45cf-de38-4d55-875f-5079037e8e6c-openshift-service-ca\") pod \"maas-api-e2e-api-a-5363ce-9c8454d5b-rxrfp\" (UID: \"563a45cf-de38-4d55-875f-5079037e8e6c\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-5363ce-9c8454d5b-rxrfp" Jul 09 16:05:41.792244 ip-10-0-133-183 kubenswrapper[2573]: I0709 16:05:41.792216 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/563a45cf-de38-4d55-875f-5079037e8e6c-maas-api-tls\") pod \"maas-api-e2e-api-a-5363ce-9c8454d5b-rxrfp\" (UID: \"563a45cf-de38-4d55-875f-5079037e8e6c\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-5363ce-9c8454d5b-rxrfp" Jul 09 16:05:41.799133 ip-10-0-133-183 kubenswrapper[2573]: I0709 16:05:41.799102 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ffgp8\" (UniqueName: \"kubernetes.io/projected/563a45cf-de38-4d55-875f-5079037e8e6c-kube-api-access-ffgp8\") pod \"maas-api-e2e-api-a-5363ce-9c8454d5b-rxrfp\" (UID: \"563a45cf-de38-4d55-875f-5079037e8e6c\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-5363ce-9c8454d5b-rxrfp" Jul 09 16:05:41.936340 ip-10-0-133-183 kubenswrapper[2573]: I0709 16:05:41.936252 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-5363ce-9c8454d5b-rxrfp" Jul 09 16:05:42.074503 ip-10-0-133-183 kubenswrapper[2573]: I0709 16:05:42.074432 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-5363ce-9c8454d5b-rxrfp"] Jul 09 16:05:42.077084 ip-10-0-133-183 kubenswrapper[2573]: W0709 16:05:42.077055 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod563a45cf_de38_4d55_875f_5079037e8e6c.slice/crio-4d308b68c3e0fb62ad08bc3f81824470e6270e93403683f082a5252f93881ec4 WatchSource:0}: Error finding container 4d308b68c3e0fb62ad08bc3f81824470e6270e93403683f082a5252f93881ec4: Status 404 returned error can't find the container with id 4d308b68c3e0fb62ad08bc3f81824470e6270e93403683f082a5252f93881ec4 Jul 09 16:05:42.277788 ip-10-0-133-183 kubenswrapper[2573]: I0709 16:05:42.277755 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-5363ce-9c8454d5b-rxrfp" event={"ID":"563a45cf-de38-4d55-875f-5079037e8e6c","Type":"ContainerStarted","Data":"4d308b68c3e0fb62ad08bc3f81824470e6270e93403683f082a5252f93881ec4"} Jul 09 16:05:45.293164 ip-10-0-133-183 kubenswrapper[2573]: I0709 16:05:45.293079 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-5363ce-9c8454d5b-rxrfp" event={"ID":"563a45cf-de38-4d55-875f-5079037e8e6c","Type":"ContainerStarted","Data":"3f2ce0db0b1180e86dee8adaebd4f531466c0e58bac5a8cebbc88c56ddbb0567"} Jul 09 16:05:45.293164 ip-10-0-133-183 kubenswrapper[2573]: I0709 16:05:45.293160 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-5363ce-9c8454d5b-rxrfp" Jul 09 16:05:45.312187 ip-10-0-133-183 kubenswrapper[2573]: I0709 16:05:45.312137 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-5363ce-9c8454d5b-rxrfp" podStartSLOduration=1.4006051990000001 podStartE2EDuration="4.312122309s" totalImagesPullingTime="2.91151711s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 16:05:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 16:05:45.310660839 +0000 UTC m=+2067.032167730" watchObservedRunningTime="2026-07-09 16:05:45.312122309 +0000 UTC m=+2067.033629177" Jul 09 16:05:49.510750 ip-10-0-133-183 kubenswrapper[2573]: I0709 16:05:49.510703 2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-fae65f-5fbf885bdc-tlcjb"] Jul 09 16:05:49.514442 ip-10-0-133-183 kubenswrapper[2573]: I0709 16:05:49.514422 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-fae65f-5fbf885bdc-tlcjb" Jul 09 16:05:49.517197 ip-10-0-133-183 kubenswrapper[2573]: I0709 16:05:49.517170 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-b-fae65f-serving-cert\"" Jul 09 16:05:49.528507 ip-10-0-133-183 kubenswrapper[2573]: I0709 16:05:49.528482 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-fae65f-5fbf885bdc-tlcjb"] Jul 09 16:05:49.553745 ip-10-0-133-183 kubenswrapper[2573]: I0709 16:05:49.553712 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/40f8d75c-1348-4d74-b2c1-949a30706242-openshift-service-ca\") pod \"maas-api-e2e-api-b-fae65f-5fbf885bdc-tlcjb\" (UID: \"40f8d75c-1348-4d74-b2c1-949a30706242\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-fae65f-5fbf885bdc-tlcjb" Jul 09 16:05:49.553745 ip-10-0-133-183 kubenswrapper[2573]: I0709 16:05:49.553747 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lxpk8\" (UniqueName: \"kubernetes.io/projected/40f8d75c-1348-4d74-b2c1-949a30706242-kube-api-access-lxpk8\") pod \"maas-api-e2e-api-b-fae65f-5fbf885bdc-tlcjb\" (UID: \"40f8d75c-1348-4d74-b2c1-949a30706242\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-fae65f-5fbf885bdc-tlcjb" Jul 09 16:05:49.554027 ip-10-0-133-183 kubenswrapper[2573]: I0709 16:05:49.553839 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/40f8d75c-1348-4d74-b2c1-949a30706242-maas-api-tls\") pod \"maas-api-e2e-api-b-fae65f-5fbf885bdc-tlcjb\" (UID: \"40f8d75c-1348-4d74-b2c1-949a30706242\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-fae65f-5fbf885bdc-tlcjb" Jul 09 16:05:49.654453 ip-10-0-133-183 kubenswrapper[2573]: I0709 16:05:49.654410 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/40f8d75c-1348-4d74-b2c1-949a30706242-openshift-service-ca\") pod \"maas-api-e2e-api-b-fae65f-5fbf885bdc-tlcjb\" (UID: \"40f8d75c-1348-4d74-b2c1-949a30706242\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-fae65f-5fbf885bdc-tlcjb" Jul 09 16:05:49.654453 ip-10-0-133-183 kubenswrapper[2573]: I0709 16:05:49.654451 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lxpk8\" (UniqueName: \"kubernetes.io/projected/40f8d75c-1348-4d74-b2c1-949a30706242-kube-api-access-lxpk8\") pod \"maas-api-e2e-api-b-fae65f-5fbf885bdc-tlcjb\" (UID: \"40f8d75c-1348-4d74-b2c1-949a30706242\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-fae65f-5fbf885bdc-tlcjb" Jul 09 16:05:49.654733 ip-10-0-133-183 kubenswrapper[2573]: I0709 16:05:49.654513 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/40f8d75c-1348-4d74-b2c1-949a30706242-maas-api-tls\") pod \"maas-api-e2e-api-b-fae65f-5fbf885bdc-tlcjb\" (UID: \"40f8d75c-1348-4d74-b2c1-949a30706242\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-fae65f-5fbf885bdc-tlcjb" Jul 09 16:05:49.654733 ip-10-0-133-183 kubenswrapper[2573]: E0709 16:05:49.654634 2573 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-api-b-fae65f-serving-cert: secret "maas-api-e2e-api-b-fae65f-serving-cert" not found Jul 09 16:05:49.654733 ip-10-0-133-183 kubenswrapper[2573]: E0709 16:05:49.654693 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/40f8d75c-1348-4d74-b2c1-949a30706242-maas-api-tls podName:40f8d75c-1348-4d74-b2c1-949a30706242 nodeName:}" failed. No retries permitted until 2026-07-09 16:05:50.154674254 +0000 UTC m=+2071.876181101 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/40f8d75c-1348-4d74-b2c1-949a30706242-maas-api-tls") pod "maas-api-e2e-api-b-fae65f-5fbf885bdc-tlcjb" (UID: "40f8d75c-1348-4d74-b2c1-949a30706242") : secret "maas-api-e2e-api-b-fae65f-serving-cert" not found Jul 09 16:05:49.655229 ip-10-0-133-183 kubenswrapper[2573]: I0709 16:05:49.655206 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/40f8d75c-1348-4d74-b2c1-949a30706242-openshift-service-ca\") pod \"maas-api-e2e-api-b-fae65f-5fbf885bdc-tlcjb\" (UID: \"40f8d75c-1348-4d74-b2c1-949a30706242\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-fae65f-5fbf885bdc-tlcjb" Jul 09 16:05:49.663931 ip-10-0-133-183 kubenswrapper[2573]: I0709 16:05:49.663870 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lxpk8\" (UniqueName: \"kubernetes.io/projected/40f8d75c-1348-4d74-b2c1-949a30706242-kube-api-access-lxpk8\") pod \"maas-api-e2e-api-b-fae65f-5fbf885bdc-tlcjb\" (UID: \"40f8d75c-1348-4d74-b2c1-949a30706242\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-fae65f-5fbf885bdc-tlcjb" Jul 09 16:05:50.159139 ip-10-0-133-183 kubenswrapper[2573]: I0709 16:05:50.159103 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/40f8d75c-1348-4d74-b2c1-949a30706242-maas-api-tls\") pod \"maas-api-e2e-api-b-fae65f-5fbf885bdc-tlcjb\" (UID: \"40f8d75c-1348-4d74-b2c1-949a30706242\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-fae65f-5fbf885bdc-tlcjb" Jul 09 16:05:50.161628 ip-10-0-133-183 kubenswrapper[2573]: I0709 16:05:50.161604 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/40f8d75c-1348-4d74-b2c1-949a30706242-maas-api-tls\") pod \"maas-api-e2e-api-b-fae65f-5fbf885bdc-tlcjb\" (UID: \"40f8d75c-1348-4d74-b2c1-949a30706242\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-fae65f-5fbf885bdc-tlcjb" Jul 09 16:05:50.425611 ip-10-0-133-183 kubenswrapper[2573]: I0709 16:05:50.425513 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-fae65f-5fbf885bdc-tlcjb" Jul 09 16:05:50.570654 ip-10-0-133-183 kubenswrapper[2573]: I0709 16:05:50.570628 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-fae65f-5fbf885bdc-tlcjb"] Jul 09 16:05:50.572720 ip-10-0-133-183 kubenswrapper[2573]: W0709 16:05:50.572685 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod40f8d75c_1348_4d74_b2c1_949a30706242.slice/crio-8d8a72fa57b0e2f0fcb334e3314eee7fee7448bf9ba33f4193de2cfa8a7c0d17 WatchSource:0}: Error finding container 8d8a72fa57b0e2f0fcb334e3314eee7fee7448bf9ba33f4193de2cfa8a7c0d17: Status 404 returned error can't find the container with id 8d8a72fa57b0e2f0fcb334e3314eee7fee7448bf9ba33f4193de2cfa8a7c0d17 Jul 09 16:05:51.304188 ip-10-0-133-183 kubenswrapper[2573]: I0709 16:05:51.304148 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-5363ce-9c8454d5b-rxrfp" Jul 09 16:05:51.321775 ip-10-0-133-183 kubenswrapper[2573]: I0709 16:05:51.321743 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-fae65f-5fbf885bdc-tlcjb" event={"ID":"40f8d75c-1348-4d74-b2c1-949a30706242","Type":"ContainerStarted","Data":"dbbf80b3bdbcf4b073076d5e01a2b209a4088d92dad910dbb1e5255a8bc362a1"} Jul 09 16:05:51.322003 ip-10-0-133-183 kubenswrapper[2573]: I0709 16:05:51.321805 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-fae65f-5fbf885bdc-tlcjb" Jul 09 16:05:51.322003 ip-10-0-133-183 kubenswrapper[2573]: I0709 16:05:51.321827 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-fae65f-5fbf885bdc-tlcjb" event={"ID":"40f8d75c-1348-4d74-b2c1-949a30706242","Type":"ContainerStarted","Data":"8d8a72fa57b0e2f0fcb334e3314eee7fee7448bf9ba33f4193de2cfa8a7c0d17"} Jul 09 16:05:51.343863 ip-10-0-133-183 kubenswrapper[2573]: I0709 16:05:51.343804 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-fae65f-5fbf885bdc-tlcjb" podStartSLOduration=1.798278394 podStartE2EDuration="2.343785544s" totalImagesPullingTime="545.50715ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 16:05:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 16:05:51.342473599 +0000 UTC m=+2073.063980468" watchObservedRunningTime="2026-07-09 16:05:51.343785544 +0000 UTC m=+2073.065292414" Jul 09 16:05:57.333448 ip-10-0-133-183 kubenswrapper[2573]: I0709 16:05:57.333420 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-fae65f-5fbf885bdc-tlcjb" Jul 09 16:06:03.215528 ip-10-0-133-183 kubenswrapper[2573]: I0709 16:06:03.215482 2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-5363ce-9c8454d5b-rxrfp"] Jul 09 16:06:03.216014 ip-10-0-133-183 kubenswrapper[2573]: I0709 16:06:03.215742 2573 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-5363ce-9c8454d5b-rxrfp" podUID="563a45cf-de38-4d55-875f-5079037e8e6c" containerName="maas-api" containerID="cri-o://3f2ce0db0b1180e86dee8adaebd4f531466c0e58bac5a8cebbc88c56ddbb0567" gracePeriod=30 Jul 09 16:06:03.379389 ip-10-0-133-183 kubenswrapper[2573]: I0709 16:06:03.379356 2573 generic.go:356] "Generic (PLEG): container finished" podID="563a45cf-de38-4d55-875f-5079037e8e6c" containerID="3f2ce0db0b1180e86dee8adaebd4f531466c0e58bac5a8cebbc88c56ddbb0567" exitCode=0 Jul 09 16:06:03.379598 ip-10-0-133-183 kubenswrapper[2573]: I0709 16:06:03.379439 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-5363ce-9c8454d5b-rxrfp" event={"ID":"563a45cf-de38-4d55-875f-5079037e8e6c","Type":"ContainerDied","Data":"3f2ce0db0b1180e86dee8adaebd4f531466c0e58bac5a8cebbc88c56ddbb0567"} Jul 09 16:06:03.462300 ip-10-0-133-183 kubenswrapper[2573]: I0709 16:06:03.462276 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-5363ce-9c8454d5b-rxrfp" Jul 09 16:06:03.583071 ip-10-0-133-183 kubenswrapper[2573]: I0709 16:06:03.582985 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/563a45cf-de38-4d55-875f-5079037e8e6c-openshift-service-ca\") pod \"563a45cf-de38-4d55-875f-5079037e8e6c\" (UID: \"563a45cf-de38-4d55-875f-5079037e8e6c\") " Jul 09 16:06:03.583071 ip-10-0-133-183 kubenswrapper[2573]: I0709 16:06:03.583022 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ffgp8\" (UniqueName: \"kubernetes.io/projected/563a45cf-de38-4d55-875f-5079037e8e6c-kube-api-access-ffgp8\") pod \"563a45cf-de38-4d55-875f-5079037e8e6c\" (UID: \"563a45cf-de38-4d55-875f-5079037e8e6c\") " Jul 09 16:06:03.583071 ip-10-0-133-183 kubenswrapper[2573]: I0709 16:06:03.583060 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/563a45cf-de38-4d55-875f-5079037e8e6c-maas-api-tls\") pod \"563a45cf-de38-4d55-875f-5079037e8e6c\" (UID: \"563a45cf-de38-4d55-875f-5079037e8e6c\") " Jul 09 16:06:03.583438 ip-10-0-133-183 kubenswrapper[2573]: I0709 16:06:03.583405 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/563a45cf-de38-4d55-875f-5079037e8e6c-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "563a45cf-de38-4d55-875f-5079037e8e6c" (UID: "563a45cf-de38-4d55-875f-5079037e8e6c"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 09 16:06:03.585143 ip-10-0-133-183 kubenswrapper[2573]: I0709 16:06:03.585117 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/563a45cf-de38-4d55-875f-5079037e8e6c-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "563a45cf-de38-4d55-875f-5079037e8e6c" (UID: "563a45cf-de38-4d55-875f-5079037e8e6c"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 09 16:06:03.585143 ip-10-0-133-183 kubenswrapper[2573]: I0709 16:06:03.585124 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/563a45cf-de38-4d55-875f-5079037e8e6c-kube-api-access-ffgp8" (OuterVolumeSpecName: "kube-api-access-ffgp8") pod "563a45cf-de38-4d55-875f-5079037e8e6c" (UID: "563a45cf-de38-4d55-875f-5079037e8e6c"). InnerVolumeSpecName "kube-api-access-ffgp8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 09 16:06:03.684596 ip-10-0-133-183 kubenswrapper[2573]: I0709 16:06:03.684559 2573 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/563a45cf-de38-4d55-875f-5079037e8e6c-openshift-service-ca\") on node \"ip-10-0-133-183.ec2.internal\" DevicePath \"\"" Jul 09 16:06:03.684596 ip-10-0-133-183 kubenswrapper[2573]: I0709 16:06:03.684591 2573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ffgp8\" (UniqueName: \"kubernetes.io/projected/563a45cf-de38-4d55-875f-5079037e8e6c-kube-api-access-ffgp8\") on node \"ip-10-0-133-183.ec2.internal\" DevicePath \"\"" Jul 09 16:06:03.684596 ip-10-0-133-183 kubenswrapper[2573]: I0709 16:06:03.684601 2573 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/563a45cf-de38-4d55-875f-5079037e8e6c-maas-api-tls\") on node \"ip-10-0-133-183.ec2.internal\" DevicePath \"\"" Jul 09 16:06:04.385861 ip-10-0-133-183 kubenswrapper[2573]: I0709 16:06:04.385830 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-5363ce-9c8454d5b-rxrfp" Jul 09 16:06:04.386289 ip-10-0-133-183 kubenswrapper[2573]: I0709 16:06:04.385860 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-5363ce-9c8454d5b-rxrfp" event={"ID":"563a45cf-de38-4d55-875f-5079037e8e6c","Type":"ContainerDied","Data":"4d308b68c3e0fb62ad08bc3f81824470e6270e93403683f082a5252f93881ec4"} Jul 09 16:06:04.386289 ip-10-0-133-183 kubenswrapper[2573]: I0709 16:06:04.385939 2573 scope.go:117] "RemoveContainer" containerID="3f2ce0db0b1180e86dee8adaebd4f531466c0e58bac5a8cebbc88c56ddbb0567" Jul 09 16:06:04.411211 ip-10-0-133-183 kubenswrapper[2573]: I0709 16:06:04.411181 2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-5363ce-9c8454d5b-rxrfp"] Jul 09 16:06:04.419070 ip-10-0-133-183 kubenswrapper[2573]: I0709 16:06:04.419042 2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-5363ce-9c8454d5b-rxrfp"] Jul 09 16:06:04.811308 ip-10-0-133-183 kubenswrapper[2573]: I0709 16:06:04.811272 2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="563a45cf-de38-4d55-875f-5079037e8e6c" path="/var/lib/kubelet/pods/563a45cf-de38-4d55-875f-5079037e8e6c/volumes" Jul 09 16:06:10.357571 ip-10-0-133-183 kubenswrapper[2573]: I0709 16:06:10.357537 2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-fae65f-5fbf885bdc-tlcjb"] Jul 09 16:06:10.358012 ip-10-0-133-183 kubenswrapper[2573]: I0709 16:06:10.357796 2573 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-fae65f-5fbf885bdc-tlcjb" podUID="40f8d75c-1348-4d74-b2c1-949a30706242" containerName="maas-api" containerID="cri-o://dbbf80b3bdbcf4b073076d5e01a2b209a4088d92dad910dbb1e5255a8bc362a1" gracePeriod=30 Jul 09 16:06:10.600145 ip-10-0-133-183 kubenswrapper[2573]: I0709 16:06:10.600121 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-fae65f-5fbf885bdc-tlcjb" Jul 09 16:06:10.752439 ip-10-0-133-183 kubenswrapper[2573]: I0709 16:06:10.752397 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/40f8d75c-1348-4d74-b2c1-949a30706242-openshift-service-ca\") pod \"40f8d75c-1348-4d74-b2c1-949a30706242\" (UID: \"40f8d75c-1348-4d74-b2c1-949a30706242\") " Jul 09 16:06:10.752618 ip-10-0-133-183 kubenswrapper[2573]: I0709 16:06:10.752452 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/40f8d75c-1348-4d74-b2c1-949a30706242-maas-api-tls\") pod \"40f8d75c-1348-4d74-b2c1-949a30706242\" (UID: \"40f8d75c-1348-4d74-b2c1-949a30706242\") " Jul 09 16:06:10.752618 ip-10-0-133-183 kubenswrapper[2573]: I0709 16:06:10.752512 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-lxpk8\" (UniqueName: \"kubernetes.io/projected/40f8d75c-1348-4d74-b2c1-949a30706242-kube-api-access-lxpk8\") pod \"40f8d75c-1348-4d74-b2c1-949a30706242\" (UID: \"40f8d75c-1348-4d74-b2c1-949a30706242\") " Jul 09 16:06:10.752791 ip-10-0-133-183 kubenswrapper[2573]: I0709 16:06:10.752767 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/40f8d75c-1348-4d74-b2c1-949a30706242-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "40f8d75c-1348-4d74-b2c1-949a30706242" (UID: "40f8d75c-1348-4d74-b2c1-949a30706242"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 09 16:06:10.754530 ip-10-0-133-183 kubenswrapper[2573]: I0709 16:06:10.754501 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/40f8d75c-1348-4d74-b2c1-949a30706242-kube-api-access-lxpk8" (OuterVolumeSpecName: "kube-api-access-lxpk8") pod "40f8d75c-1348-4d74-b2c1-949a30706242" (UID: "40f8d75c-1348-4d74-b2c1-949a30706242"). InnerVolumeSpecName "kube-api-access-lxpk8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 09 16:06:10.754530 ip-10-0-133-183 kubenswrapper[2573]: I0709 16:06:10.754515 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/40f8d75c-1348-4d74-b2c1-949a30706242-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "40f8d75c-1348-4d74-b2c1-949a30706242" (UID: "40f8d75c-1348-4d74-b2c1-949a30706242"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 09 16:06:10.854004 ip-10-0-133-183 kubenswrapper[2573]: I0709 16:06:10.853965 2573 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/40f8d75c-1348-4d74-b2c1-949a30706242-openshift-service-ca\") on node \"ip-10-0-133-183.ec2.internal\" DevicePath \"\"" Jul 09 16:06:10.854004 ip-10-0-133-183 kubenswrapper[2573]: I0709 16:06:10.853994 2573 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/40f8d75c-1348-4d74-b2c1-949a30706242-maas-api-tls\") on node \"ip-10-0-133-183.ec2.internal\" DevicePath \"\"" Jul 09 16:06:10.854004 ip-10-0-133-183 kubenswrapper[2573]: I0709 16:06:10.854003 2573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lxpk8\" (UniqueName: \"kubernetes.io/projected/40f8d75c-1348-4d74-b2c1-949a30706242-kube-api-access-lxpk8\") on node \"ip-10-0-133-183.ec2.internal\" DevicePath \"\"" Jul 09 16:06:11.427765 ip-10-0-133-183 kubenswrapper[2573]: I0709 16:06:11.427726 2573 generic.go:356] "Generic (PLEG): container finished" podID="40f8d75c-1348-4d74-b2c1-949a30706242" containerID="dbbf80b3bdbcf4b073076d5e01a2b209a4088d92dad910dbb1e5255a8bc362a1" exitCode=0 Jul 09 16:06:11.428257 ip-10-0-133-183 kubenswrapper[2573]: I0709 16:06:11.427769 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-fae65f-5fbf885bdc-tlcjb" Jul 09 16:06:11.428257 ip-10-0-133-183 kubenswrapper[2573]: I0709 16:06:11.427812 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-fae65f-5fbf885bdc-tlcjb" event={"ID":"40f8d75c-1348-4d74-b2c1-949a30706242","Type":"ContainerDied","Data":"dbbf80b3bdbcf4b073076d5e01a2b209a4088d92dad910dbb1e5255a8bc362a1"} Jul 09 16:06:11.428257 ip-10-0-133-183 kubenswrapper[2573]: I0709 16:06:11.427867 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-fae65f-5fbf885bdc-tlcjb" event={"ID":"40f8d75c-1348-4d74-b2c1-949a30706242","Type":"ContainerDied","Data":"8d8a72fa57b0e2f0fcb334e3314eee7fee7448bf9ba33f4193de2cfa8a7c0d17"} Jul 09 16:06:11.428257 ip-10-0-133-183 kubenswrapper[2573]: I0709 16:06:11.427911 2573 scope.go:117] "RemoveContainer" containerID="dbbf80b3bdbcf4b073076d5e01a2b209a4088d92dad910dbb1e5255a8bc362a1" Jul 09 16:06:11.436357 ip-10-0-133-183 kubenswrapper[2573]: I0709 16:06:11.436340 2573 scope.go:117] "RemoveContainer" containerID="dbbf80b3bdbcf4b073076d5e01a2b209a4088d92dad910dbb1e5255a8bc362a1" Jul 09 16:06:11.436623 ip-10-0-133-183 kubenswrapper[2573]: E0709 16:06:11.436607 2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"dbbf80b3bdbcf4b073076d5e01a2b209a4088d92dad910dbb1e5255a8bc362a1\": container with ID starting with dbbf80b3bdbcf4b073076d5e01a2b209a4088d92dad910dbb1e5255a8bc362a1 not found: ID does not exist" containerID="dbbf80b3bdbcf4b073076d5e01a2b209a4088d92dad910dbb1e5255a8bc362a1" Jul 09 16:06:11.436687 ip-10-0-133-183 kubenswrapper[2573]: I0709 16:06:11.436633 2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dbbf80b3bdbcf4b073076d5e01a2b209a4088d92dad910dbb1e5255a8bc362a1"} err="failed to get container status \"dbbf80b3bdbcf4b073076d5e01a2b209a4088d92dad910dbb1e5255a8bc362a1\": rpc error: code = NotFound desc = could not find container \"dbbf80b3bdbcf4b073076d5e01a2b209a4088d92dad910dbb1e5255a8bc362a1\": container with ID starting with dbbf80b3bdbcf4b073076d5e01a2b209a4088d92dad910dbb1e5255a8bc362a1 not found: ID does not exist" Jul 09 16:06:11.458176 ip-10-0-133-183 kubenswrapper[2573]: I0709 16:06:11.458148 2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-fae65f-5fbf885bdc-tlcjb"] Jul 09 16:06:11.472629 ip-10-0-133-183 kubenswrapper[2573]: I0709 16:06:11.472599 2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-fae65f-5fbf885bdc-tlcjb"] Jul 09 16:06:12.810274 ip-10-0-133-183 kubenswrapper[2573]: I0709 16:06:12.810233 2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="40f8d75c-1348-4d74-b2c1-949a30706242" path="/var/lib/kubelet/pods/40f8d75c-1348-4d74-b2c1-949a30706242/volumes" Jul 09 16:06:18.830081 ip-10-0-133-183 kubenswrapper[2573]: I0709 16:06:18.829964 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-b7b9v_db22f88d-a953-488e-b384-67f991351f73/ovn-acl-logging/0.log" Jul 09 16:06:18.858799 ip-10-0-133-183 kubenswrapper[2573]: I0709 16:06:18.842703 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-b7b9v_db22f88d-a953-488e-b384-67f991351f73/ovn-acl-logging/0.log" Jul 09 16:06:20.691329 ip-10-0-133-183 kubenswrapper[2573]: I0709 16:06:20.691297 2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-c8e6dd-5b964cc97b-wtvc5"] Jul 09 16:06:20.691911 ip-10-0-133-183 kubenswrapper[2573]: I0709 16:06:20.691741 2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="563a45cf-de38-4d55-875f-5079037e8e6c" containerName="maas-api" Jul 09 16:06:20.691911 ip-10-0-133-183 kubenswrapper[2573]: I0709 16:06:20.691755 2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="563a45cf-de38-4d55-875f-5079037e8e6c" containerName="maas-api" Jul 09 16:06:20.691911 ip-10-0-133-183 kubenswrapper[2573]: I0709 16:06:20.691773 2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="40f8d75c-1348-4d74-b2c1-949a30706242" containerName="maas-api" Jul 09 16:06:20.691911 ip-10-0-133-183 kubenswrapper[2573]: I0709 16:06:20.691779 2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="40f8d75c-1348-4d74-b2c1-949a30706242" containerName="maas-api" Jul 09 16:06:20.691911 ip-10-0-133-183 kubenswrapper[2573]: I0709 16:06:20.691839 2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="40f8d75c-1348-4d74-b2c1-949a30706242" containerName="maas-api" Jul 09 16:06:20.691911 ip-10-0-133-183 kubenswrapper[2573]: I0709 16:06:20.691847 2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="563a45cf-de38-4d55-875f-5079037e8e6c" containerName="maas-api" Jul 09 16:06:20.696490 ip-10-0-133-183 kubenswrapper[2573]: I0709 16:06:20.696466 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-c8e6dd-5b964cc97b-wtvc5" Jul 09 16:06:20.698967 ip-10-0-133-183 kubenswrapper[2573]: I0709 16:06:20.698946 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-a-c8e6dd-serving-cert\"" Jul 09 16:06:20.707828 ip-10-0-133-183 kubenswrapper[2573]: I0709 16:06:20.707800 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-c8e6dd-5b964cc97b-wtvc5"] Jul 09 16:06:20.843233 ip-10-0-133-183 kubenswrapper[2573]: I0709 16:06:20.843186 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f789839a-4bf2-4a77-82bf-05a61111fd17-openshift-service-ca\") pod \"maas-api-e2e-shared-a-c8e6dd-5b964cc97b-wtvc5\" (UID: \"f789839a-4bf2-4a77-82bf-05a61111fd17\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-c8e6dd-5b964cc97b-wtvc5" Jul 09 16:06:20.843437 ip-10-0-133-183 kubenswrapper[2573]: I0709 16:06:20.843278 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f789839a-4bf2-4a77-82bf-05a61111fd17-maas-api-tls\") pod \"maas-api-e2e-shared-a-c8e6dd-5b964cc97b-wtvc5\" (UID: \"f789839a-4bf2-4a77-82bf-05a61111fd17\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-c8e6dd-5b964cc97b-wtvc5" Jul 09 16:06:20.843437 ip-10-0-133-183 kubenswrapper[2573]: I0709 16:06:20.843306 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fdr87\" (UniqueName: \"kubernetes.io/projected/f789839a-4bf2-4a77-82bf-05a61111fd17-kube-api-access-fdr87\") pod \"maas-api-e2e-shared-a-c8e6dd-5b964cc97b-wtvc5\" (UID: \"f789839a-4bf2-4a77-82bf-05a61111fd17\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-c8e6dd-5b964cc97b-wtvc5" Jul 09 16:06:20.944787 ip-10-0-133-183 kubenswrapper[2573]: I0709 16:06:20.944750 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f789839a-4bf2-4a77-82bf-05a61111fd17-openshift-service-ca\") pod \"maas-api-e2e-shared-a-c8e6dd-5b964cc97b-wtvc5\" (UID: \"f789839a-4bf2-4a77-82bf-05a61111fd17\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-c8e6dd-5b964cc97b-wtvc5" Jul 09 16:06:20.944971 ip-10-0-133-183 kubenswrapper[2573]: I0709 16:06:20.944847 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f789839a-4bf2-4a77-82bf-05a61111fd17-maas-api-tls\") pod \"maas-api-e2e-shared-a-c8e6dd-5b964cc97b-wtvc5\" (UID: \"f789839a-4bf2-4a77-82bf-05a61111fd17\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-c8e6dd-5b964cc97b-wtvc5" Jul 09 16:06:20.944971 ip-10-0-133-183 kubenswrapper[2573]: I0709 16:06:20.944905 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fdr87\" (UniqueName: \"kubernetes.io/projected/f789839a-4bf2-4a77-82bf-05a61111fd17-kube-api-access-fdr87\") pod \"maas-api-e2e-shared-a-c8e6dd-5b964cc97b-wtvc5\" (UID: \"f789839a-4bf2-4a77-82bf-05a61111fd17\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-c8e6dd-5b964cc97b-wtvc5" Jul 09 16:06:20.945505 ip-10-0-133-183 kubenswrapper[2573]: I0709 16:06:20.945488 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f789839a-4bf2-4a77-82bf-05a61111fd17-openshift-service-ca\") pod \"maas-api-e2e-shared-a-c8e6dd-5b964cc97b-wtvc5\" (UID: \"f789839a-4bf2-4a77-82bf-05a61111fd17\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-c8e6dd-5b964cc97b-wtvc5" Jul 09 16:06:20.947325 ip-10-0-133-183 kubenswrapper[2573]: I0709 16:06:20.947303 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f789839a-4bf2-4a77-82bf-05a61111fd17-maas-api-tls\") pod \"maas-api-e2e-shared-a-c8e6dd-5b964cc97b-wtvc5\" (UID: \"f789839a-4bf2-4a77-82bf-05a61111fd17\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-c8e6dd-5b964cc97b-wtvc5" Jul 09 16:06:20.953741 ip-10-0-133-183 kubenswrapper[2573]: I0709 16:06:20.953717 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fdr87\" (UniqueName: \"kubernetes.io/projected/f789839a-4bf2-4a77-82bf-05a61111fd17-kube-api-access-fdr87\") pod \"maas-api-e2e-shared-a-c8e6dd-5b964cc97b-wtvc5\" (UID: \"f789839a-4bf2-4a77-82bf-05a61111fd17\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-c8e6dd-5b964cc97b-wtvc5" Jul 09 16:06:21.006679 ip-10-0-133-183 kubenswrapper[2573]: I0709 16:06:21.006638 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-c8e6dd-5b964cc97b-wtvc5" Jul 09 16:06:21.145598 ip-10-0-133-183 kubenswrapper[2573]: I0709 16:06:21.145573 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-c8e6dd-5b964cc97b-wtvc5"] Jul 09 16:06:21.148530 ip-10-0-133-183 kubenswrapper[2573]: W0709 16:06:21.148500 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf789839a_4bf2_4a77_82bf_05a61111fd17.slice/crio-f99d672c8054f7f8a81d33c9e2db303cac4f046f6e5fe89ae369212413695f48 WatchSource:0}: Error finding container f99d672c8054f7f8a81d33c9e2db303cac4f046f6e5fe89ae369212413695f48: Status 404 returned error can't find the container with id f99d672c8054f7f8a81d33c9e2db303cac4f046f6e5fe89ae369212413695f48 Jul 09 16:06:21.479977 ip-10-0-133-183 kubenswrapper[2573]: I0709 16:06:21.479942 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-c8e6dd-5b964cc97b-wtvc5" event={"ID":"f789839a-4bf2-4a77-82bf-05a61111fd17","Type":"ContainerStarted","Data":"f99d672c8054f7f8a81d33c9e2db303cac4f046f6e5fe89ae369212413695f48"} Jul 09 16:06:22.486013 ip-10-0-133-183 kubenswrapper[2573]: I0709 16:06:22.485978 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-c8e6dd-5b964cc97b-wtvc5" event={"ID":"f789839a-4bf2-4a77-82bf-05a61111fd17","Type":"ContainerStarted","Data":"c9df3542ae983b5faa490f8ca035c466035278bc73feb9ae5bbed7a260ad19c0"} Jul 09 16:06:22.486430 ip-10-0-133-183 kubenswrapper[2573]: I0709 16:06:22.486100 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-c8e6dd-5b964cc97b-wtvc5" Jul 09 16:06:22.505964 ip-10-0-133-183 kubenswrapper[2573]: I0709 16:06:22.505911 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-c8e6dd-5b964cc97b-wtvc5" podStartSLOduration=2.099655245 podStartE2EDuration="2.505893106s" totalImagesPullingTime="406.237861ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 16:06:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 16:06:22.50456551 +0000 UTC m=+2104.226072402" watchObservedRunningTime="2026-07-09 16:06:22.505893106 +0000 UTC m=+2104.227399964" Jul 09 16:06:28.497080 ip-10-0-133-183 kubenswrapper[2573]: I0709 16:06:28.497048 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-c8e6dd-5b964cc97b-wtvc5" Jul 09 16:06:28.781261 ip-10-0-133-183 kubenswrapper[2573]: I0709 16:06:28.781177 2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-1a4b78-6fd884c5dd-nktdh"] Jul 09 16:06:28.784809 ip-10-0-133-183 kubenswrapper[2573]: I0709 16:06:28.784787 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-1a4b78-6fd884c5dd-nktdh" Jul 09 16:06:28.787448 ip-10-0-133-183 kubenswrapper[2573]: I0709 16:06:28.787428 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-b-1a4b78-serving-cert\"" Jul 09 16:06:28.802760 ip-10-0-133-183 kubenswrapper[2573]: I0709 16:06:28.802738 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-1a4b78-6fd884c5dd-nktdh"] Jul 09 16:06:28.919907 ip-10-0-133-183 kubenswrapper[2573]: I0709 16:06:28.919856 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c0769b8d-7773-4955-bff7-a21abc7788ce-openshift-service-ca\") pod \"maas-api-e2e-shared-b-1a4b78-6fd884c5dd-nktdh\" (UID: \"c0769b8d-7773-4955-bff7-a21abc7788ce\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-1a4b78-6fd884c5dd-nktdh" Jul 09 16:06:28.920097 ip-10-0-133-183 kubenswrapper[2573]: I0709 16:06:28.919937 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z4lk4\" (UniqueName: \"kubernetes.io/projected/c0769b8d-7773-4955-bff7-a21abc7788ce-kube-api-access-z4lk4\") pod \"maas-api-e2e-shared-b-1a4b78-6fd884c5dd-nktdh\" (UID: \"c0769b8d-7773-4955-bff7-a21abc7788ce\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-1a4b78-6fd884c5dd-nktdh" Jul 09 16:06:28.920097 ip-10-0-133-183 kubenswrapper[2573]: I0709 16:06:28.920004 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c0769b8d-7773-4955-bff7-a21abc7788ce-maas-api-tls\") pod \"maas-api-e2e-shared-b-1a4b78-6fd884c5dd-nktdh\" (UID: \"c0769b8d-7773-4955-bff7-a21abc7788ce\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-1a4b78-6fd884c5dd-nktdh" Jul 09 16:06:29.020896 ip-10-0-133-183 kubenswrapper[2573]: I0709 16:06:29.020848 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c0769b8d-7773-4955-bff7-a21abc7788ce-openshift-service-ca\") pod \"maas-api-e2e-shared-b-1a4b78-6fd884c5dd-nktdh\" (UID: \"c0769b8d-7773-4955-bff7-a21abc7788ce\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-1a4b78-6fd884c5dd-nktdh" Jul 09 16:06:29.021060 ip-10-0-133-183 kubenswrapper[2573]: I0709 16:06:29.020923 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z4lk4\" (UniqueName: \"kubernetes.io/projected/c0769b8d-7773-4955-bff7-a21abc7788ce-kube-api-access-z4lk4\") pod \"maas-api-e2e-shared-b-1a4b78-6fd884c5dd-nktdh\" (UID: \"c0769b8d-7773-4955-bff7-a21abc7788ce\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-1a4b78-6fd884c5dd-nktdh" Jul 09 16:06:29.021060 ip-10-0-133-183 kubenswrapper[2573]: I0709 16:06:29.020980 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c0769b8d-7773-4955-bff7-a21abc7788ce-maas-api-tls\") pod \"maas-api-e2e-shared-b-1a4b78-6fd884c5dd-nktdh\" (UID: \"c0769b8d-7773-4955-bff7-a21abc7788ce\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-1a4b78-6fd884c5dd-nktdh" Jul 09 16:06:29.021469 ip-10-0-133-183 kubenswrapper[2573]: I0709 16:06:29.021447 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c0769b8d-7773-4955-bff7-a21abc7788ce-openshift-service-ca\") pod \"maas-api-e2e-shared-b-1a4b78-6fd884c5dd-nktdh\" (UID: \"c0769b8d-7773-4955-bff7-a21abc7788ce\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-1a4b78-6fd884c5dd-nktdh" Jul 09 16:06:29.023244 ip-10-0-133-183 kubenswrapper[2573]: I0709 16:06:29.023222 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c0769b8d-7773-4955-bff7-a21abc7788ce-maas-api-tls\") pod \"maas-api-e2e-shared-b-1a4b78-6fd884c5dd-nktdh\" (UID: \"c0769b8d-7773-4955-bff7-a21abc7788ce\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-1a4b78-6fd884c5dd-nktdh" Jul 09 16:06:29.032679 ip-10-0-133-183 kubenswrapper[2573]: I0709 16:06:29.032624 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-z4lk4\" (UniqueName: \"kubernetes.io/projected/c0769b8d-7773-4955-bff7-a21abc7788ce-kube-api-access-z4lk4\") pod \"maas-api-e2e-shared-b-1a4b78-6fd884c5dd-nktdh\" (UID: \"c0769b8d-7773-4955-bff7-a21abc7788ce\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-1a4b78-6fd884c5dd-nktdh" Jul 09 16:06:29.095716 ip-10-0-133-183 kubenswrapper[2573]: I0709 16:06:29.095681 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-1a4b78-6fd884c5dd-nktdh" Jul 09 16:06:29.237186 ip-10-0-133-183 kubenswrapper[2573]: I0709 16:06:29.237157 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-1a4b78-6fd884c5dd-nktdh"] Jul 09 16:06:29.239771 ip-10-0-133-183 kubenswrapper[2573]: W0709 16:06:29.239745 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc0769b8d_7773_4955_bff7_a21abc7788ce.slice/crio-756bad3b02c0ee769210556e5ae085618a1d502b33bd96f3260ca5c3e039d8c0 WatchSource:0}: Error finding container 756bad3b02c0ee769210556e5ae085618a1d502b33bd96f3260ca5c3e039d8c0: Status 404 returned error can't find the container with id 756bad3b02c0ee769210556e5ae085618a1d502b33bd96f3260ca5c3e039d8c0 Jul 09 16:06:29.520941 ip-10-0-133-183 kubenswrapper[2573]: I0709 16:06:29.520906 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-1a4b78-6fd884c5dd-nktdh" event={"ID":"c0769b8d-7773-4955-bff7-a21abc7788ce","Type":"ContainerStarted","Data":"756bad3b02c0ee769210556e5ae085618a1d502b33bd96f3260ca5c3e039d8c0"} Jul 09 16:06:32.535993 ip-10-0-133-183 kubenswrapper[2573]: I0709 16:06:32.535953 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-1a4b78-6fd884c5dd-nktdh" event={"ID":"c0769b8d-7773-4955-bff7-a21abc7788ce","Type":"ContainerStarted","Data":"5317506786b5e07a0f951b3bd422bdd5d8f86d6b83bbdc4ecf399d2d3fe5d410"} Jul 09 16:06:32.536418 ip-10-0-133-183 kubenswrapper[2573]: I0709 16:06:32.536078 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-1a4b78-6fd884c5dd-nktdh" Jul 09 16:06:32.557404 ip-10-0-133-183 kubenswrapper[2573]: I0709 16:06:32.557358 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-1a4b78-6fd884c5dd-nktdh" podStartSLOduration=2.143870121 podStartE2EDuration="4.557343106s" totalImagesPullingTime="2.413472985s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 16:06:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 16:06:32.555378665 +0000 UTC m=+2114.276885551" watchObservedRunningTime="2026-07-09 16:06:32.557343106 +0000 UTC m=+2114.278849973" Jul 09 16:06:38.546426 ip-10-0-133-183 kubenswrapper[2573]: I0709 16:06:38.546396 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-1a4b78-6fd884c5dd-nktdh" Jul 09 16:09:39.191377 ip-10-0-133-183 kubenswrapper[2573]: I0709 16:09:39.191340 2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-c8e6dd-5b964cc97b-wtvc5"] Jul 09 16:09:39.192211 ip-10-0-133-183 kubenswrapper[2573]: I0709 16:09:39.192150 2573 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-c8e6dd-5b964cc97b-wtvc5" podUID="f789839a-4bf2-4a77-82bf-05a61111fd17" containerName="maas-api" containerID="cri-o://c9df3542ae983b5faa490f8ca035c466035278bc73feb9ae5bbed7a260ad19c0" gracePeriod=30 Jul 09 16:09:39.412937 ip-10-0-133-183 kubenswrapper[2573]: I0709 16:09:39.412871 2573 generic.go:356] "Generic (PLEG): container finished" podID="f789839a-4bf2-4a77-82bf-05a61111fd17" containerID="c9df3542ae983b5faa490f8ca035c466035278bc73feb9ae5bbed7a260ad19c0" exitCode=0 Jul 09 16:09:39.412937 ip-10-0-133-183 kubenswrapper[2573]: I0709 16:09:39.412909 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-c8e6dd-5b964cc97b-wtvc5" event={"ID":"f789839a-4bf2-4a77-82bf-05a61111fd17","Type":"ContainerDied","Data":"c9df3542ae983b5faa490f8ca035c466035278bc73feb9ae5bbed7a260ad19c0"} Jul 09 16:09:39.436980 ip-10-0-133-183 kubenswrapper[2573]: I0709 16:09:39.436956 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-c8e6dd-5b964cc97b-wtvc5" Jul 09 16:09:39.567642 ip-10-0-133-183 kubenswrapper[2573]: I0709 16:09:39.567604 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f789839a-4bf2-4a77-82bf-05a61111fd17-maas-api-tls\") pod \"f789839a-4bf2-4a77-82bf-05a61111fd17\" (UID: \"f789839a-4bf2-4a77-82bf-05a61111fd17\") " Jul 09 16:09:39.567844 ip-10-0-133-183 kubenswrapper[2573]: I0709 16:09:39.567692 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-fdr87\" (UniqueName: \"kubernetes.io/projected/f789839a-4bf2-4a77-82bf-05a61111fd17-kube-api-access-fdr87\") pod \"f789839a-4bf2-4a77-82bf-05a61111fd17\" (UID: \"f789839a-4bf2-4a77-82bf-05a61111fd17\") " Jul 09 16:09:39.567844 ip-10-0-133-183 kubenswrapper[2573]: I0709 16:09:39.567719 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f789839a-4bf2-4a77-82bf-05a61111fd17-openshift-service-ca\") pod \"f789839a-4bf2-4a77-82bf-05a61111fd17\" (UID: \"f789839a-4bf2-4a77-82bf-05a61111fd17\") " Jul 09 16:09:39.568136 ip-10-0-133-183 kubenswrapper[2573]: I0709 16:09:39.568104 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f789839a-4bf2-4a77-82bf-05a61111fd17-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "f789839a-4bf2-4a77-82bf-05a61111fd17" (UID: "f789839a-4bf2-4a77-82bf-05a61111fd17"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 09 16:09:39.569803 ip-10-0-133-183 kubenswrapper[2573]: I0709 16:09:39.569782 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f789839a-4bf2-4a77-82bf-05a61111fd17-kube-api-access-fdr87" (OuterVolumeSpecName: "kube-api-access-fdr87") pod "f789839a-4bf2-4a77-82bf-05a61111fd17" (UID: "f789839a-4bf2-4a77-82bf-05a61111fd17"). InnerVolumeSpecName "kube-api-access-fdr87". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 09 16:09:39.569894 ip-10-0-133-183 kubenswrapper[2573]: I0709 16:09:39.569820 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f789839a-4bf2-4a77-82bf-05a61111fd17-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "f789839a-4bf2-4a77-82bf-05a61111fd17" (UID: "f789839a-4bf2-4a77-82bf-05a61111fd17"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 09 16:09:39.668660 ip-10-0-133-183 kubenswrapper[2573]: I0709 16:09:39.668621 2573 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f789839a-4bf2-4a77-82bf-05a61111fd17-maas-api-tls\") on node \"ip-10-0-133-183.ec2.internal\" DevicePath \"\"" Jul 09 16:09:39.668660 ip-10-0-133-183 kubenswrapper[2573]: I0709 16:09:39.668653 2573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fdr87\" (UniqueName: \"kubernetes.io/projected/f789839a-4bf2-4a77-82bf-05a61111fd17-kube-api-access-fdr87\") on node \"ip-10-0-133-183.ec2.internal\" DevicePath \"\"" Jul 09 16:09:39.668660 ip-10-0-133-183 kubenswrapper[2573]: I0709 16:09:39.668666 2573 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f789839a-4bf2-4a77-82bf-05a61111fd17-openshift-service-ca\") on node \"ip-10-0-133-183.ec2.internal\" DevicePath \"\"" Jul 09 16:09:40.419523 ip-10-0-133-183 kubenswrapper[2573]: I0709 16:09:40.419499 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-c8e6dd-5b964cc97b-wtvc5" Jul 09 16:09:40.419974 ip-10-0-133-183 kubenswrapper[2573]: I0709 16:09:40.419492 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-c8e6dd-5b964cc97b-wtvc5" event={"ID":"f789839a-4bf2-4a77-82bf-05a61111fd17","Type":"ContainerDied","Data":"f99d672c8054f7f8a81d33c9e2db303cac4f046f6e5fe89ae369212413695f48"} Jul 09 16:09:40.419974 ip-10-0-133-183 kubenswrapper[2573]: I0709 16:09:40.419628 2573 scope.go:117] "RemoveContainer" containerID="c9df3542ae983b5faa490f8ca035c466035278bc73feb9ae5bbed7a260ad19c0" Jul 09 16:09:40.441999 ip-10-0-133-183 kubenswrapper[2573]: I0709 16:09:40.441974 2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-c8e6dd-5b964cc97b-wtvc5"] Jul 09 16:09:40.446672 ip-10-0-133-183 kubenswrapper[2573]: I0709 16:09:40.446648 2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-c8e6dd-5b964cc97b-wtvc5"] Jul 09 16:09:40.810993 ip-10-0-133-183 kubenswrapper[2573]: I0709 16:09:40.810961 2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f789839a-4bf2-4a77-82bf-05a61111fd17" path="/var/lib/kubelet/pods/f789839a-4bf2-4a77-82bf-05a61111fd17/volumes" Jul 09 16:09:46.474036 ip-10-0-133-183 kubenswrapper[2573]: I0709 16:09:46.474007 2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-1a4b78-6fd884c5dd-nktdh"] Jul 09 16:09:46.474430 ip-10-0-133-183 kubenswrapper[2573]: I0709 16:09:46.474208 2573 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-1a4b78-6fd884c5dd-nktdh" podUID="c0769b8d-7773-4955-bff7-a21abc7788ce" containerName="maas-api" containerID="cri-o://5317506786b5e07a0f951b3bd422bdd5d8f86d6b83bbdc4ecf399d2d3fe5d410" gracePeriod=30 Jul 09 16:09:46.718887 ip-10-0-133-183 kubenswrapper[2573]: I0709 16:09:46.718850 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-1a4b78-6fd884c5dd-nktdh" Jul 09 16:09:46.839184 ip-10-0-133-183 kubenswrapper[2573]: I0709 16:09:46.839100 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-z4lk4\" (UniqueName: \"kubernetes.io/projected/c0769b8d-7773-4955-bff7-a21abc7788ce-kube-api-access-z4lk4\") pod \"c0769b8d-7773-4955-bff7-a21abc7788ce\" (UID: \"c0769b8d-7773-4955-bff7-a21abc7788ce\") " Jul 09 16:09:46.839184 ip-10-0-133-183 kubenswrapper[2573]: I0709 16:09:46.839133 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c0769b8d-7773-4955-bff7-a21abc7788ce-maas-api-tls\") pod \"c0769b8d-7773-4955-bff7-a21abc7788ce\" (UID: \"c0769b8d-7773-4955-bff7-a21abc7788ce\") " Jul 09 16:09:46.839415 ip-10-0-133-183 kubenswrapper[2573]: I0709 16:09:46.839196 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c0769b8d-7773-4955-bff7-a21abc7788ce-openshift-service-ca\") pod \"c0769b8d-7773-4955-bff7-a21abc7788ce\" (UID: \"c0769b8d-7773-4955-bff7-a21abc7788ce\") " Jul 09 16:09:46.839630 ip-10-0-133-183 kubenswrapper[2573]: I0709 16:09:46.839605 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c0769b8d-7773-4955-bff7-a21abc7788ce-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "c0769b8d-7773-4955-bff7-a21abc7788ce" (UID: "c0769b8d-7773-4955-bff7-a21abc7788ce"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 09 16:09:46.841283 ip-10-0-133-183 kubenswrapper[2573]: I0709 16:09:46.841252 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c0769b8d-7773-4955-bff7-a21abc7788ce-kube-api-access-z4lk4" (OuterVolumeSpecName: "kube-api-access-z4lk4") pod "c0769b8d-7773-4955-bff7-a21abc7788ce" (UID: "c0769b8d-7773-4955-bff7-a21abc7788ce"). InnerVolumeSpecName "kube-api-access-z4lk4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 09 16:09:46.841283 ip-10-0-133-183 kubenswrapper[2573]: I0709 16:09:46.841263 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c0769b8d-7773-4955-bff7-a21abc7788ce-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "c0769b8d-7773-4955-bff7-a21abc7788ce" (UID: "c0769b8d-7773-4955-bff7-a21abc7788ce"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 09 16:09:46.939778 ip-10-0-133-183 kubenswrapper[2573]: I0709 16:09:46.939742 2573 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c0769b8d-7773-4955-bff7-a21abc7788ce-openshift-service-ca\") on node \"ip-10-0-133-183.ec2.internal\" DevicePath \"\"" Jul 09 16:09:46.939778 ip-10-0-133-183 kubenswrapper[2573]: I0709 16:09:46.939769 2573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z4lk4\" (UniqueName: \"kubernetes.io/projected/c0769b8d-7773-4955-bff7-a21abc7788ce-kube-api-access-z4lk4\") on node \"ip-10-0-133-183.ec2.internal\" DevicePath \"\"" Jul 09 16:09:46.939778 ip-10-0-133-183 kubenswrapper[2573]: I0709 16:09:46.939780 2573 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c0769b8d-7773-4955-bff7-a21abc7788ce-maas-api-tls\") on node \"ip-10-0-133-183.ec2.internal\" DevicePath \"\"" Jul 09 16:09:47.463100 ip-10-0-133-183 kubenswrapper[2573]: I0709 16:09:47.463071 2573 generic.go:356] "Generic (PLEG): container finished" podID="c0769b8d-7773-4955-bff7-a21abc7788ce" containerID="5317506786b5e07a0f951b3bd422bdd5d8f86d6b83bbdc4ecf399d2d3fe5d410" exitCode=0 Jul 09 16:09:47.463260 ip-10-0-133-183 kubenswrapper[2573]: I0709 16:09:47.463129 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-1a4b78-6fd884c5dd-nktdh" Jul 09 16:09:47.463260 ip-10-0-133-183 kubenswrapper[2573]: I0709 16:09:47.463148 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-1a4b78-6fd884c5dd-nktdh" event={"ID":"c0769b8d-7773-4955-bff7-a21abc7788ce","Type":"ContainerDied","Data":"5317506786b5e07a0f951b3bd422bdd5d8f86d6b83bbdc4ecf399d2d3fe5d410"} Jul 09 16:09:47.463260 ip-10-0-133-183 kubenswrapper[2573]: I0709 16:09:47.463194 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-1a4b78-6fd884c5dd-nktdh" event={"ID":"c0769b8d-7773-4955-bff7-a21abc7788ce","Type":"ContainerDied","Data":"756bad3b02c0ee769210556e5ae085618a1d502b33bd96f3260ca5c3e039d8c0"} Jul 09 16:09:47.463260 ip-10-0-133-183 kubenswrapper[2573]: I0709 16:09:47.463218 2573 scope.go:117] "RemoveContainer" containerID="5317506786b5e07a0f951b3bd422bdd5d8f86d6b83bbdc4ecf399d2d3fe5d410" Jul 09 16:09:47.472418 ip-10-0-133-183 kubenswrapper[2573]: I0709 16:09:47.472395 2573 scope.go:117] "RemoveContainer" containerID="5317506786b5e07a0f951b3bd422bdd5d8f86d6b83bbdc4ecf399d2d3fe5d410" Jul 09 16:09:47.472694 ip-10-0-133-183 kubenswrapper[2573]: E0709 16:09:47.472674 2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5317506786b5e07a0f951b3bd422bdd5d8f86d6b83bbdc4ecf399d2d3fe5d410\": container with ID starting with 5317506786b5e07a0f951b3bd422bdd5d8f86d6b83bbdc4ecf399d2d3fe5d410 not found: ID does not exist" containerID="5317506786b5e07a0f951b3bd422bdd5d8f86d6b83bbdc4ecf399d2d3fe5d410" Jul 09 16:09:47.472762 ip-10-0-133-183 kubenswrapper[2573]: I0709 16:09:47.472710 2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5317506786b5e07a0f951b3bd422bdd5d8f86d6b83bbdc4ecf399d2d3fe5d410"} err="failed to get container status \"5317506786b5e07a0f951b3bd422bdd5d8f86d6b83bbdc4ecf399d2d3fe5d410\": rpc error: code = NotFound desc = could not find container \"5317506786b5e07a0f951b3bd422bdd5d8f86d6b83bbdc4ecf399d2d3fe5d410\": container with ID starting with 5317506786b5e07a0f951b3bd422bdd5d8f86d6b83bbdc4ecf399d2d3fe5d410 not found: ID does not exist" Jul 09 16:09:47.488428 ip-10-0-133-183 kubenswrapper[2573]: I0709 16:09:47.488400 2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-1a4b78-6fd884c5dd-nktdh"] Jul 09 16:09:47.500656 ip-10-0-133-183 kubenswrapper[2573]: I0709 16:09:47.500620 2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-1a4b78-6fd884c5dd-nktdh"] Jul 09 16:09:48.811716 ip-10-0-133-183 kubenswrapper[2573]: I0709 16:09:48.811678 2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c0769b8d-7773-4955-bff7-a21abc7788ce" path="/var/lib/kubelet/pods/c0769b8d-7773-4955-bff7-a21abc7788ce/volumes" Jul 09 16:09:58.734724 ip-10-0-133-183 kubenswrapper[2573]: I0709 16:09:58.734690 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-647758b7f9-bgl5w_6af05785-c895-415e-923d-23ee28c4eece/manager/0.log" Jul 09 16:09:59.265575 ip-10-0-133-183 kubenswrapper[2573]: I0709 16:09:59.265548 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-9bb557cf6-lmk4b_d038f9a0-b62d-4b40-bbb0-a758173d50c0/copy-manifests/0.log" Jul 09 16:09:59.278137 ip-10-0-133-183 kubenswrapper[2573]: I0709 16:09:59.278114 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-9bb557cf6-lmk4b_d038f9a0-b62d-4b40-bbb0-a758173d50c0/manager/0.log" Jul 09 16:10:02.194314 ip-10-0-133-183 kubenswrapper[2573]: I0709 16:10:02.194283 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-94bb8fbfd-68ctz_10ae1018-c000-41b2-bd1c-e933573ae519/discovery/0.log" Jul 09 16:10:02.697090 ip-10-0-133-183 kubenswrapper[2573]: I0709 16:10:02.697062 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-7964457696-m5gbg_c7bca669-f5f2-41b2-87be-4ff0a7e25993/payload-pre-processing/0.log" Jul 09 16:10:02.827961 ip-10-0-133-183 kubenswrapper[2573]: I0709 16:10:02.827930 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-58f86c7798-jkj5d_c99251e1-0ac9-4fe6-9acf-e099c6bcafdc/payload-processing/0.log" Jul 09 16:10:03.320832 ip-10-0-133-183 kubenswrapper[2573]: I0709 16:10:03.320802 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-distinct-2-simulated-kserve-67d8d5d6d-bkcwb_69646e64-e81d-4dd0-99c4-408a27cc16b7/main/0.log" Jul 09 16:10:03.445241 ip-10-0-133-183 kubenswrapper[2573]: I0709 16:10:03.445209 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-distinct-simulated-kserve-59975d69f9-zfnbz_feacfd9b-35b5-40bf-88cf-8ff322bde482/main/0.log" Jul 09 16:10:12.803127 ip-10-0-133-183 kubenswrapper[2573]: I0709 16:10:12.803050 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-h2cs6_d9ea5934-356a-47e5-9fc6-1ae4f39f4e19/global-pull-secret-syncer/0.log" Jul 09 16:10:12.953685 ip-10-0-133-183 kubenswrapper[2573]: I0709 16:10:12.953651 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-zrrxr_02be15c9-9800-4cae-9293-de04d5c845f8/konnectivity-agent/0.log" Jul 09 16:10:13.036528 ip-10-0-133-183 kubenswrapper[2573]: I0709 16:10:13.036497 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-133-183.ec2.internal_06227558853f1ac07df9b5c4930c43af/haproxy/0.log" Jul 09 16:10:19.635493 ip-10-0-133-183 kubenswrapper[2573]: I0709 16:10:19.635411 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_d060b445-2d60-481c-a52d-0fd1c8025564/alertmanager/0.log" Jul 09 16:10:19.733349 ip-10-0-133-183 kubenswrapper[2573]: I0709 16:10:19.733318 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_d060b445-2d60-481c-a52d-0fd1c8025564/config-reloader/0.log" Jul 09 16:10:19.780535 ip-10-0-133-183 kubenswrapper[2573]: I0709 16:10:19.780511 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_d060b445-2d60-481c-a52d-0fd1c8025564/kube-rbac-proxy-web/0.log" Jul 09 16:10:19.812475 ip-10-0-133-183 kubenswrapper[2573]: I0709 16:10:19.812449 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_d060b445-2d60-481c-a52d-0fd1c8025564/kube-rbac-proxy/0.log" Jul 09 16:10:19.840685 ip-10-0-133-183 kubenswrapper[2573]: I0709 16:10:19.840662 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_d060b445-2d60-481c-a52d-0fd1c8025564/kube-rbac-proxy-metric/0.log" Jul 09 16:10:19.870870 ip-10-0-133-183 kubenswrapper[2573]: I0709 16:10:19.870844 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_d060b445-2d60-481c-a52d-0fd1c8025564/prom-label-proxy/0.log" Jul 09 16:10:19.904101 ip-10-0-133-183 kubenswrapper[2573]: I0709 16:10:19.904028 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_d060b445-2d60-481c-a52d-0fd1c8025564/init-config-reloader/0.log" Jul 09 16:10:20.080684 ip-10-0-133-183 kubenswrapper[2573]: I0709 16:10:20.080655 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_metrics-server-68cc7b6dfc-pqtq8_b98b23c1-1d45-4d6b-83cf-579c250605b7/metrics-server/0.log" Jul 09 16:10:20.110318 ip-10-0-133-183 kubenswrapper[2573]: I0709 16:10:20.110290 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_monitoring-plugin-5f68df7f58-47x4v_07b98dec-386e-4afc-9345-8e8c2c6f9642/monitoring-plugin/0.log" Jul 09 16:10:20.146755 ip-10-0-133-183 kubenswrapper[2573]: I0709 16:10:20.146725 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-4jbr9_1886ffd8-a444-41ce-803c-58c58160faff/node-exporter/0.log" Jul 09 16:10:20.174072 ip-10-0-133-183 kubenswrapper[2573]: I0709 16:10:20.173997 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-4jbr9_1886ffd8-a444-41ce-803c-58c58160faff/kube-rbac-proxy/0.log" Jul 09 16:10:20.204499 ip-10-0-133-183 kubenswrapper[2573]: I0709 16:10:20.204470 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-4jbr9_1886ffd8-a444-41ce-803c-58c58160faff/init-textfile/0.log" Jul 09 16:10:20.413332 ip-10-0-133-183 kubenswrapper[2573]: I0709 16:10:20.413304 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-84bd68b896-q9km7_9889ca25-29ca-4226-8c5a-4e2eb8040868/kube-rbac-proxy-main/0.log" Jul 09 16:10:20.441011 ip-10-0-133-183 kubenswrapper[2573]: I0709 16:10:20.440932 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-84bd68b896-q9km7_9889ca25-29ca-4226-8c5a-4e2eb8040868/kube-rbac-proxy-self/0.log" Jul 09 16:10:20.470803 ip-10-0-133-183 kubenswrapper[2573]: I0709 16:10:20.470775 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-84bd68b896-q9km7_9889ca25-29ca-4226-8c5a-4e2eb8040868/openshift-state-metrics/0.log" Jul 09 16:10:20.515188 ip-10-0-133-183 kubenswrapper[2573]: I0709 16:10:20.515154 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_c219e7c1-044b-4e08-8455-362bec98f934/prometheus/0.log" Jul 09 16:10:20.546938 ip-10-0-133-183 kubenswrapper[2573]: I0709 16:10:20.546913 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_c219e7c1-044b-4e08-8455-362bec98f934/config-reloader/0.log" Jul 09 16:10:20.582566 ip-10-0-133-183 kubenswrapper[2573]: I0709 16:10:20.582541 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_c219e7c1-044b-4e08-8455-362bec98f934/thanos-sidecar/0.log" Jul 09 16:10:20.609164 ip-10-0-133-183 kubenswrapper[2573]: I0709 16:10:20.609133 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_c219e7c1-044b-4e08-8455-362bec98f934/kube-rbac-proxy-web/0.log" Jul 09 16:10:20.637502 ip-10-0-133-183 kubenswrapper[2573]: I0709 16:10:20.637474 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_c219e7c1-044b-4e08-8455-362bec98f934/kube-rbac-proxy/0.log" Jul 09 16:10:20.665432 ip-10-0-133-183 kubenswrapper[2573]: I0709 16:10:20.665409 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_c219e7c1-044b-4e08-8455-362bec98f934/kube-rbac-proxy-thanos/0.log" Jul 09 16:10:20.692050 ip-10-0-133-183 kubenswrapper[2573]: I0709 16:10:20.691973 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_c219e7c1-044b-4e08-8455-362bec98f934/init-config-reloader/0.log" Jul 09 16:10:20.724751 ip-10-0-133-183 kubenswrapper[2573]: I0709 16:10:20.724721 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-64785b6675-smrzt_b7e1c1ed-4b4d-4062-b4d2-d9f3759d4756/prometheus-operator/0.log" Jul 09 16:10:20.749536 ip-10-0-133-183 kubenswrapper[2573]: I0709 16:10:20.749509 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-64785b6675-smrzt_b7e1c1ed-4b4d-4062-b4d2-d9f3759d4756/kube-rbac-proxy/0.log" Jul 09 16:10:22.222957 ip-10-0-133-183 kubenswrapper[2573]: I0709 16:10:22.222920 2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-9qrm8/perf-node-gather-daemonset-vwrrh"] Jul 09 16:10:22.223346 ip-10-0-133-183 kubenswrapper[2573]: I0709 16:10:22.223281 2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c0769b8d-7773-4955-bff7-a21abc7788ce" containerName="maas-api" Jul 09 16:10:22.223346 ip-10-0-133-183 kubenswrapper[2573]: I0709 16:10:22.223291 2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="c0769b8d-7773-4955-bff7-a21abc7788ce" containerName="maas-api" Jul 09 16:10:22.223346 ip-10-0-133-183 kubenswrapper[2573]: I0709 16:10:22.223315 2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f789839a-4bf2-4a77-82bf-05a61111fd17" containerName="maas-api" Jul 09 16:10:22.223346 ip-10-0-133-183 kubenswrapper[2573]: I0709 16:10:22.223320 2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="f789839a-4bf2-4a77-82bf-05a61111fd17" containerName="maas-api" Jul 09 16:10:22.223523 ip-10-0-133-183 kubenswrapper[2573]: I0709 16:10:22.223374 2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="f789839a-4bf2-4a77-82bf-05a61111fd17" containerName="maas-api" Jul 09 16:10:22.223523 ip-10-0-133-183 kubenswrapper[2573]: I0709 16:10:22.223382 2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="c0769b8d-7773-4955-bff7-a21abc7788ce" containerName="maas-api" Jul 09 16:10:22.227930 ip-10-0-133-183 kubenswrapper[2573]: I0709 16:10:22.227906 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-9qrm8/perf-node-gather-daemonset-vwrrh" Jul 09 16:10:22.230654 ip-10-0-133-183 kubenswrapper[2573]: I0709 16:10:22.230632 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-9qrm8\"/\"kube-root-ca.crt\"" Jul 09 16:10:22.230831 ip-10-0-133-183 kubenswrapper[2573]: I0709 16:10:22.230791 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-9qrm8\"/\"openshift-service-ca.crt\"" Jul 09 16:10:22.231644 ip-10-0-133-183 kubenswrapper[2573]: I0709 16:10:22.231621 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-9qrm8\"/\"default-dockercfg-xvqd8\"" Jul 09 16:10:22.236406 ip-10-0-133-183 kubenswrapper[2573]: I0709 16:10:22.236386 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-9qrm8/perf-node-gather-daemonset-vwrrh"] Jul 09 16:10:22.270153 ip-10-0-133-183 kubenswrapper[2573]: I0709 16:10:22.270120 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/3dcfef4c-2522-41d3-9d0f-9853abe52fae-proc\") pod \"perf-node-gather-daemonset-vwrrh\" (UID: \"3dcfef4c-2522-41d3-9d0f-9853abe52fae\") " pod="openshift-must-gather-9qrm8/perf-node-gather-daemonset-vwrrh" Jul 09 16:10:22.270153 ip-10-0-133-183 kubenswrapper[2573]: I0709 16:10:22.270154 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/3dcfef4c-2522-41d3-9d0f-9853abe52fae-lib-modules\") pod \"perf-node-gather-daemonset-vwrrh\" (UID: \"3dcfef4c-2522-41d3-9d0f-9853abe52fae\") " pod="openshift-must-gather-9qrm8/perf-node-gather-daemonset-vwrrh" Jul 09 16:10:22.270401 ip-10-0-133-183 kubenswrapper[2573]: I0709 16:10:22.270177 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/3dcfef4c-2522-41d3-9d0f-9853abe52fae-podres\") pod \"perf-node-gather-daemonset-vwrrh\" (UID: \"3dcfef4c-2522-41d3-9d0f-9853abe52fae\") " pod="openshift-must-gather-9qrm8/perf-node-gather-daemonset-vwrrh" Jul 09 16:10:22.270401 ip-10-0-133-183 kubenswrapper[2573]: I0709 16:10:22.270313 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rz55z\" (UniqueName: \"kubernetes.io/projected/3dcfef4c-2522-41d3-9d0f-9853abe52fae-kube-api-access-rz55z\") pod \"perf-node-gather-daemonset-vwrrh\" (UID: \"3dcfef4c-2522-41d3-9d0f-9853abe52fae\") " pod="openshift-must-gather-9qrm8/perf-node-gather-daemonset-vwrrh" Jul 09 16:10:22.270401 ip-10-0-133-183 kubenswrapper[2573]: I0709 16:10:22.270352 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/3dcfef4c-2522-41d3-9d0f-9853abe52fae-sys\") pod \"perf-node-gather-daemonset-vwrrh\" (UID: \"3dcfef4c-2522-41d3-9d0f-9853abe52fae\") " pod="openshift-must-gather-9qrm8/perf-node-gather-daemonset-vwrrh" Jul 09 16:10:22.371225 ip-10-0-133-183 kubenswrapper[2573]: I0709 16:10:22.371180 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/3dcfef4c-2522-41d3-9d0f-9853abe52fae-podres\") pod \"perf-node-gather-daemonset-vwrrh\" (UID: \"3dcfef4c-2522-41d3-9d0f-9853abe52fae\") " pod="openshift-must-gather-9qrm8/perf-node-gather-daemonset-vwrrh" Jul 09 16:10:22.371404 ip-10-0-133-183 kubenswrapper[2573]: I0709 16:10:22.371286 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rz55z\" (UniqueName: \"kubernetes.io/projected/3dcfef4c-2522-41d3-9d0f-9853abe52fae-kube-api-access-rz55z\") pod \"perf-node-gather-daemonset-vwrrh\" (UID: \"3dcfef4c-2522-41d3-9d0f-9853abe52fae\") " pod="openshift-must-gather-9qrm8/perf-node-gather-daemonset-vwrrh" Jul 09 16:10:22.371404 ip-10-0-133-183 kubenswrapper[2573]: I0709 16:10:22.371319 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/3dcfef4c-2522-41d3-9d0f-9853abe52fae-sys\") pod \"perf-node-gather-daemonset-vwrrh\" (UID: \"3dcfef4c-2522-41d3-9d0f-9853abe52fae\") " pod="openshift-must-gather-9qrm8/perf-node-gather-daemonset-vwrrh" Jul 09 16:10:22.371404 ip-10-0-133-183 kubenswrapper[2573]: I0709 16:10:22.371365 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/3dcfef4c-2522-41d3-9d0f-9853abe52fae-podres\") pod \"perf-node-gather-daemonset-vwrrh\" (UID: \"3dcfef4c-2522-41d3-9d0f-9853abe52fae\") " pod="openshift-must-gather-9qrm8/perf-node-gather-daemonset-vwrrh" Jul 09 16:10:22.371404 ip-10-0-133-183 kubenswrapper[2573]: I0709 16:10:22.371388 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/3dcfef4c-2522-41d3-9d0f-9853abe52fae-proc\") pod \"perf-node-gather-daemonset-vwrrh\" (UID: \"3dcfef4c-2522-41d3-9d0f-9853abe52fae\") " pod="openshift-must-gather-9qrm8/perf-node-gather-daemonset-vwrrh" Jul 09 16:10:22.371632 ip-10-0-133-183 kubenswrapper[2573]: I0709 16:10:22.371410 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/3dcfef4c-2522-41d3-9d0f-9853abe52fae-lib-modules\") pod \"perf-node-gather-daemonset-vwrrh\" (UID: \"3dcfef4c-2522-41d3-9d0f-9853abe52fae\") " pod="openshift-must-gather-9qrm8/perf-node-gather-daemonset-vwrrh" Jul 09 16:10:22.371632 ip-10-0-133-183 kubenswrapper[2573]: I0709 16:10:22.371421 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/3dcfef4c-2522-41d3-9d0f-9853abe52fae-sys\") pod \"perf-node-gather-daemonset-vwrrh\" (UID: \"3dcfef4c-2522-41d3-9d0f-9853abe52fae\") " pod="openshift-must-gather-9qrm8/perf-node-gather-daemonset-vwrrh" Jul 09 16:10:22.371632 ip-10-0-133-183 kubenswrapper[2573]: I0709 16:10:22.371506 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/3dcfef4c-2522-41d3-9d0f-9853abe52fae-proc\") pod \"perf-node-gather-daemonset-vwrrh\" (UID: \"3dcfef4c-2522-41d3-9d0f-9853abe52fae\") " pod="openshift-must-gather-9qrm8/perf-node-gather-daemonset-vwrrh" Jul 09 16:10:22.371632 ip-10-0-133-183 kubenswrapper[2573]: I0709 16:10:22.371564 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/3dcfef4c-2522-41d3-9d0f-9853abe52fae-lib-modules\") pod \"perf-node-gather-daemonset-vwrrh\" (UID: \"3dcfef4c-2522-41d3-9d0f-9853abe52fae\") " pod="openshift-must-gather-9qrm8/perf-node-gather-daemonset-vwrrh" Jul 09 16:10:22.381666 ip-10-0-133-183 kubenswrapper[2573]: I0709 16:10:22.381632 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rz55z\" (UniqueName: \"kubernetes.io/projected/3dcfef4c-2522-41d3-9d0f-9853abe52fae-kube-api-access-rz55z\") pod \"perf-node-gather-daemonset-vwrrh\" (UID: \"3dcfef4c-2522-41d3-9d0f-9853abe52fae\") " pod="openshift-must-gather-9qrm8/perf-node-gather-daemonset-vwrrh" Jul 09 16:10:22.539767 ip-10-0-133-183 kubenswrapper[2573]: I0709 16:10:22.539670 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-9qrm8/perf-node-gather-daemonset-vwrrh" Jul 09 16:10:22.675092 ip-10-0-133-183 kubenswrapper[2573]: I0709 16:10:22.675059 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-9qrm8/perf-node-gather-daemonset-vwrrh"] Jul 09 16:10:22.679171 ip-10-0-133-183 kubenswrapper[2573]: W0709 16:10:22.679132 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod3dcfef4c_2522_41d3_9d0f_9853abe52fae.slice/crio-b88ab4239f978e26780220cf64752d1d8d15348df4a16d6b1d7abd227ffa6bac WatchSource:0}: Error finding container b88ab4239f978e26780220cf64752d1d8d15348df4a16d6b1d7abd227ffa6bac: Status 404 returned error can't find the container with id b88ab4239f978e26780220cf64752d1d8d15348df4a16d6b1d7abd227ffa6bac Jul 09 16:10:23.631744 ip-10-0-133-183 kubenswrapper[2573]: I0709 16:10:23.631708 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-9qrm8/perf-node-gather-daemonset-vwrrh" event={"ID":"3dcfef4c-2522-41d3-9d0f-9853abe52fae","Type":"ContainerStarted","Data":"9126ec9fcf822bbf23a00f7c2fce6998b17b327c732d6ac5a9199630a8e9664a"} Jul 09 16:10:23.632158 ip-10-0-133-183 kubenswrapper[2573]: I0709 16:10:23.631760 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-9qrm8/perf-node-gather-daemonset-vwrrh" event={"ID":"3dcfef4c-2522-41d3-9d0f-9853abe52fae","Type":"ContainerStarted","Data":"b88ab4239f978e26780220cf64752d1d8d15348df4a16d6b1d7abd227ffa6bac"} Jul 09 16:10:23.632158 ip-10-0-133-183 kubenswrapper[2573]: I0709 16:10:23.631832 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-9qrm8/perf-node-gather-daemonset-vwrrh" Jul 09 16:10:23.655451 ip-10-0-133-183 kubenswrapper[2573]: I0709 16:10:23.655394 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-9qrm8/perf-node-gather-daemonset-vwrrh" podStartSLOduration=1.655377673 podStartE2EDuration="1.655377673s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-09 16:10:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 16:10:23.653854148 +0000 UTC m=+2345.375361017" watchObservedRunningTime="2026-07-09 16:10:23.655377673 +0000 UTC m=+2345.376884540" Jul 09 16:10:25.121768 ip-10-0-133-183 kubenswrapper[2573]: I0709 16:10:25.121727 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-ffqv6_b9e43e24-97dc-464d-a445-2100e1b546a9/dns/0.log" Jul 09 16:10:25.147911 ip-10-0-133-183 kubenswrapper[2573]: I0709 16:10:25.147857 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-ffqv6_b9e43e24-97dc-464d-a445-2100e1b546a9/kube-rbac-proxy/0.log" Jul 09 16:10:25.264726 ip-10-0-133-183 kubenswrapper[2573]: I0709 16:10:25.264701 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-b5c8h_9ad54b5e-c82e-480c-814f-e1ab55bf32fb/dns-node-resolver/0.log" Jul 09 16:10:25.939491 ip-10-0-133-183 kubenswrapper[2573]: I0709 16:10:25.939465 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-trnjd_1094a821-3b3e-46d8-ac90-64c7ae6e8cad/node-ca/0.log" Jul 09 16:10:27.027990 ip-10-0-133-183 kubenswrapper[2573]: I0709 16:10:27.027957 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-94bb8fbfd-68ctz_10ae1018-c000-41b2-bd1c-e933573ae519/discovery/0.log" Jul 09 16:10:27.189295 ip-10-0-133-183 kubenswrapper[2573]: I0709 16:10:27.189251 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-7964457696-m5gbg_c7bca669-f5f2-41b2-87be-4ff0a7e25993/payload-pre-processing/0.log" Jul 09 16:10:27.236456 ip-10-0-133-183 kubenswrapper[2573]: I0709 16:10:27.236423 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-58f86c7798-jkj5d_c99251e1-0ac9-4fe6-9acf-e099c6bcafdc/payload-processing/0.log" Jul 09 16:10:27.966564 ip-10-0-133-183 kubenswrapper[2573]: I0709 16:10:27.966536 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-fnfp8_d4b4e91f-6d12-45a0-a779-3fcda0e75f08/serve-healthcheck-canary/0.log" Jul 09 16:10:28.490561 ip-10-0-133-183 kubenswrapper[2573]: I0709 16:10:28.490496 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-5kk64_8c2fc9e6-25b9-4d1d-9ea0-c6de5f1367e2/kube-rbac-proxy/0.log" Jul 09 16:10:28.516243 ip-10-0-133-183 kubenswrapper[2573]: I0709 16:10:28.516217 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-5kk64_8c2fc9e6-25b9-4d1d-9ea0-c6de5f1367e2/exporter/0.log" Jul 09 16:10:28.540830 ip-10-0-133-183 kubenswrapper[2573]: I0709 16:10:28.540806 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-5kk64_8c2fc9e6-25b9-4d1d-9ea0-c6de5f1367e2/extractor/0.log" Jul 09 16:10:29.647050 ip-10-0-133-183 kubenswrapper[2573]: I0709 16:10:29.647022 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-9qrm8/perf-node-gather-daemonset-vwrrh" Jul 09 16:10:31.194784 ip-10-0-133-183 kubenswrapper[2573]: I0709 16:10:31.190845 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-647758b7f9-bgl5w_6af05785-c895-415e-923d-23ee28c4eece/manager/0.log" Jul 09 16:10:31.417630 ip-10-0-133-183 kubenswrapper[2573]: I0709 16:10:31.417549 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-9bb557cf6-lmk4b_d038f9a0-b62d-4b40-bbb0-a758173d50c0/manager/0.log" Jul 09 16:10:31.443008 ip-10-0-133-183 kubenswrapper[2573]: I0709 16:10:31.442982 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-9bb557cf6-lmk4b_d038f9a0-b62d-4b40-bbb0-a758173d50c0/copy-manifests/0.log" Jul 09 16:10:32.243705 ip-10-0-133-183 kubenswrapper[2573]: I0709 16:10:32.243677 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-94bb8fbfd-68ctz_10ae1018-c000-41b2-bd1c-e933573ae519/discovery/0.log" Jul 09 16:10:32.360629 ip-10-0-133-183 kubenswrapper[2573]: I0709 16:10:32.360596 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-7964457696-m5gbg_c7bca669-f5f2-41b2-87be-4ff0a7e25993/payload-pre-processing/0.log" Jul 09 16:10:32.397688 ip-10-0-133-183 kubenswrapper[2573]: I0709 16:10:32.397663 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-58f86c7798-jkj5d_c99251e1-0ac9-4fe6-9acf-e099c6bcafdc/payload-processing/0.log" Jul 09 16:10:32.838064 ip-10-0-133-183 kubenswrapper[2573]: I0709 16:10:32.838033 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-lws-operator_lws-controller-manager-5c8fc68c98-rzldz_0cbf8218-3798-438b-836c-7d5e99be8137/manager/0.log" Jul 09 16:10:32.894709 ip-10-0-133-183 kubenswrapper[2573]: I0709 16:10:32.894684 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-lws-operator_openshift-lws-operator-fd8ccff4c-kqf2g_f0a24581-6eee-4f3a-adad-476ed7659833/openshift-lws-operator/0.log" Jul 09 16:10:33.668940 ip-10-0-133-183 kubenswrapper[2573]: I0709 16:10:33.668845 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-94bb8fbfd-68ctz_10ae1018-c000-41b2-bd1c-e933573ae519/discovery/0.log" Jul 09 16:10:33.803721 ip-10-0-133-183 kubenswrapper[2573]: I0709 16:10:33.803690 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-7964457696-m5gbg_c7bca669-f5f2-41b2-87be-4ff0a7e25993/payload-pre-processing/0.log" Jul 09 16:10:33.841583 ip-10-0-133-183 kubenswrapper[2573]: I0709 16:10:33.841551 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-58f86c7798-jkj5d_c99251e1-0ac9-4fe6-9acf-e099c6bcafdc/payload-processing/0.log" Jul 09 16:10:37.852071 ip-10-0-133-183 kubenswrapper[2573]: I0709 16:10:37.852038 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-7f59b54f6c-vj7jk_f4b336f8-f3d0-463d-952c-13622744cf40/migrator/0.log" Jul 09 16:10:37.893655 ip-10-0-133-183 kubenswrapper[2573]: I0709 16:10:37.893628 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-7f59b54f6c-vj7jk_f4b336f8-f3d0-463d-952c-13622744cf40/graceful-termination/0.log"