Jul 13 22:11:18.641707 ip-10-0-133-179 systemd[1]: kubelet.service: Failed to load environment files: No such file or directory Jul 13 22:11:18.641719 ip-10-0-133-179 systemd[1]: kubelet.service: Failed to run 'start-pre' task: No such file or directory Jul 13 22:11:18.641728 ip-10-0-133-179 systemd[1]: kubelet.service: Failed with result 'resources'. Jul 13 22:11:18.641946 ip-10-0-133-179 systemd[1]: Failed to start Kubernetes Kubelet. Jul 13 22:11:28.737509 ip-10-0-133-179 systemd[1]: kubelet.service: Failed to schedule restart job: Unit crio.service not found. Jul 13 22:11:28.737529 ip-10-0-133-179 systemd[1]: kubelet.service: Failed with result 'resources'. -- Boot 0d8ac52856e341dbbf3ee078c50b36b9 -- Jul 13 22:13:23.031556 ip-10-0-133-179 systemd[1]: Starting Kubernetes Kubelet... Jul 13 22:13:25.133275 ip-10-0-133-179 kubenswrapper[2583]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jul 13 22:13:25.133275 ip-10-0-133-179 kubenswrapper[2583]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version. Jul 13 22:13:25.133275 ip-10-0-133-179 kubenswrapper[2583]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jul 13 22:13:25.133275 ip-10-0-133-179 kubenswrapper[2583]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI. Jul 13 22:13:25.133275 ip-10-0-133-179 kubenswrapper[2583]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jul 13 22:13:25.133275 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:13:23.480716 2583 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime" Jul 13 22:13:25.133275 ip-10-0-133-179 kubenswrapper[2583]: W0713 22:13:23.486312 2583 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 13 22:13:25.133275 ip-10-0-133-179 kubenswrapper[2583]: W0713 22:13:23.486325 2583 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 13 22:13:25.133275 ip-10-0-133-179 kubenswrapper[2583]: W0713 22:13:23.486329 2583 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 13 22:13:25.133275 ip-10-0-133-179 kubenswrapper[2583]: W0713 22:13:23.486332 2583 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 13 22:13:25.133275 ip-10-0-133-179 kubenswrapper[2583]: W0713 22:13:23.486335 2583 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 13 22:13:25.133275 ip-10-0-133-179 kubenswrapper[2583]: W0713 22:13:23.486338 2583 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 13 22:13:25.133275 ip-10-0-133-179 kubenswrapper[2583]: W0713 22:13:23.486341 2583 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 13 22:13:25.133275 ip-10-0-133-179 kubenswrapper[2583]: W0713 22:13:23.486344 2583 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 13 22:13:25.135523 ip-10-0-133-179 kubenswrapper[2583]: W0713 22:13:23.486347 2583 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 13 22:13:25.135523 ip-10-0-133-179 kubenswrapper[2583]: W0713 22:13:23.486350 2583 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 13 22:13:25.135523 ip-10-0-133-179 kubenswrapper[2583]: W0713 22:13:23.486356 2583 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 13 22:13:25.135523 ip-10-0-133-179 kubenswrapper[2583]: W0713 22:13:23.486360 2583 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 13 22:13:25.135523 ip-10-0-133-179 kubenswrapper[2583]: W0713 22:13:23.486363 2583 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 13 22:13:25.135523 ip-10-0-133-179 kubenswrapper[2583]: W0713 22:13:23.486366 2583 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 13 22:13:25.135523 ip-10-0-133-179 kubenswrapper[2583]: W0713 22:13:23.486369 2583 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 13 22:13:25.135523 ip-10-0-133-179 kubenswrapper[2583]: W0713 22:13:23.486371 2583 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 13 22:13:25.135523 ip-10-0-133-179 kubenswrapper[2583]: W0713 22:13:23.486374 2583 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 13 22:13:25.135523 ip-10-0-133-179 kubenswrapper[2583]: W0713 22:13:23.486384 2583 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 13 22:13:25.135523 ip-10-0-133-179 kubenswrapper[2583]: W0713 22:13:23.486387 2583 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 13 22:13:25.135523 ip-10-0-133-179 kubenswrapper[2583]: W0713 22:13:23.486390 2583 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 13 22:13:25.135523 ip-10-0-133-179 kubenswrapper[2583]: W0713 22:13:23.486392 2583 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 13 22:13:25.135523 ip-10-0-133-179 kubenswrapper[2583]: W0713 22:13:23.486395 2583 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 13 22:13:25.135523 ip-10-0-133-179 kubenswrapper[2583]: W0713 22:13:23.486398 2583 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 13 22:13:25.135523 ip-10-0-133-179 kubenswrapper[2583]: W0713 22:13:23.486401 2583 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 13 22:13:25.135523 ip-10-0-133-179 kubenswrapper[2583]: W0713 22:13:23.486405 2583 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 13 22:13:25.135523 ip-10-0-133-179 kubenswrapper[2583]: W0713 22:13:23.486408 2583 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 13 22:13:25.135523 ip-10-0-133-179 kubenswrapper[2583]: W0713 22:13:23.486410 2583 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 13 22:13:25.136157 ip-10-0-133-179 kubenswrapper[2583]: W0713 22:13:23.486413 2583 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 13 22:13:25.136157 ip-10-0-133-179 kubenswrapper[2583]: W0713 22:13:23.486416 2583 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 13 22:13:25.136157 ip-10-0-133-179 kubenswrapper[2583]: W0713 22:13:23.486419 2583 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 13 22:13:25.136157 ip-10-0-133-179 kubenswrapper[2583]: W0713 22:13:23.486421 2583 feature_gate.go:328] unrecognized feature gate: Example2 Jul 13 22:13:25.136157 ip-10-0-133-179 kubenswrapper[2583]: W0713 22:13:23.486424 2583 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 13 22:13:25.136157 ip-10-0-133-179 kubenswrapper[2583]: W0713 22:13:23.486427 2583 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 13 22:13:25.136157 ip-10-0-133-179 kubenswrapper[2583]: W0713 22:13:23.486431 2583 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 13 22:13:25.136157 ip-10-0-133-179 kubenswrapper[2583]: W0713 22:13:23.486434 2583 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 13 22:13:25.136157 ip-10-0-133-179 kubenswrapper[2583]: W0713 22:13:23.486436 2583 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 13 22:13:25.136157 ip-10-0-133-179 kubenswrapper[2583]: W0713 22:13:23.486440 2583 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 13 22:13:25.136157 ip-10-0-133-179 kubenswrapper[2583]: W0713 22:13:23.486443 2583 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 13 22:13:25.136157 ip-10-0-133-179 kubenswrapper[2583]: W0713 22:13:23.486447 2583 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 13 22:13:25.136157 ip-10-0-133-179 kubenswrapper[2583]: W0713 22:13:23.486451 2583 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 13 22:13:25.136157 ip-10-0-133-179 kubenswrapper[2583]: W0713 22:13:23.486453 2583 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 13 22:13:25.136157 ip-10-0-133-179 kubenswrapper[2583]: W0713 22:13:23.486456 2583 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 13 22:13:25.136157 ip-10-0-133-179 kubenswrapper[2583]: W0713 22:13:23.486459 2583 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 13 22:13:25.136157 ip-10-0-133-179 kubenswrapper[2583]: W0713 22:13:23.486463 2583 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 13 22:13:25.136157 ip-10-0-133-179 kubenswrapper[2583]: W0713 22:13:23.486466 2583 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 13 22:13:25.136157 ip-10-0-133-179 kubenswrapper[2583]: W0713 22:13:23.486468 2583 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 13 22:13:25.137388 ip-10-0-133-179 kubenswrapper[2583]: W0713 22:13:23.486472 2583 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 13 22:13:25.137388 ip-10-0-133-179 kubenswrapper[2583]: W0713 22:13:23.486475 2583 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 13 22:13:25.137388 ip-10-0-133-179 kubenswrapper[2583]: W0713 22:13:23.486478 2583 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 13 22:13:25.137388 ip-10-0-133-179 kubenswrapper[2583]: W0713 22:13:23.486481 2583 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 13 22:13:25.137388 ip-10-0-133-179 kubenswrapper[2583]: W0713 22:13:23.486484 2583 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 13 22:13:25.137388 ip-10-0-133-179 kubenswrapper[2583]: W0713 22:13:23.486487 2583 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 13 22:13:25.137388 ip-10-0-133-179 kubenswrapper[2583]: W0713 22:13:23.486489 2583 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 13 22:13:25.137388 ip-10-0-133-179 kubenswrapper[2583]: W0713 22:13:23.486492 2583 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 13 22:13:25.137388 ip-10-0-133-179 kubenswrapper[2583]: W0713 22:13:23.486495 2583 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 13 22:13:25.137388 ip-10-0-133-179 kubenswrapper[2583]: W0713 22:13:23.486497 2583 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 13 22:13:25.137388 ip-10-0-133-179 kubenswrapper[2583]: W0713 22:13:23.486500 2583 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 13 22:13:25.137388 ip-10-0-133-179 kubenswrapper[2583]: W0713 22:13:23.486503 2583 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 13 22:13:25.137388 ip-10-0-133-179 kubenswrapper[2583]: W0713 22:13:23.486505 2583 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 13 22:13:25.137388 ip-10-0-133-179 kubenswrapper[2583]: W0713 22:13:23.486508 2583 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 13 22:13:25.137388 ip-10-0-133-179 kubenswrapper[2583]: W0713 22:13:23.486511 2583 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 13 22:13:25.137388 ip-10-0-133-179 kubenswrapper[2583]: W0713 22:13:23.486513 2583 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 13 22:13:25.137388 ip-10-0-133-179 kubenswrapper[2583]: W0713 22:13:23.486516 2583 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 13 22:13:25.137388 ip-10-0-133-179 kubenswrapper[2583]: W0713 22:13:23.486519 2583 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 13 22:13:25.137388 ip-10-0-133-179 kubenswrapper[2583]: W0713 22:13:23.486522 2583 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 13 22:13:25.137388 ip-10-0-133-179 kubenswrapper[2583]: W0713 22:13:23.486524 2583 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 13 22:13:25.138008 ip-10-0-133-179 kubenswrapper[2583]: W0713 22:13:23.486527 2583 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 13 22:13:25.138008 ip-10-0-133-179 kubenswrapper[2583]: W0713 22:13:23.486530 2583 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 13 22:13:25.138008 ip-10-0-133-179 kubenswrapper[2583]: W0713 22:13:23.486532 2583 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 13 22:13:25.138008 ip-10-0-133-179 kubenswrapper[2583]: W0713 22:13:23.486535 2583 feature_gate.go:328] unrecognized feature gate: Example Jul 13 22:13:25.138008 ip-10-0-133-179 kubenswrapper[2583]: W0713 22:13:23.486538 2583 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 13 22:13:25.138008 ip-10-0-133-179 kubenswrapper[2583]: W0713 22:13:23.486540 2583 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 13 22:13:25.138008 ip-10-0-133-179 kubenswrapper[2583]: W0713 22:13:23.486543 2583 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 13 22:13:25.138008 ip-10-0-133-179 kubenswrapper[2583]: W0713 22:13:23.486546 2583 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 13 22:13:25.138008 ip-10-0-133-179 kubenswrapper[2583]: W0713 22:13:23.486549 2583 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 13 22:13:25.138008 ip-10-0-133-179 kubenswrapper[2583]: W0713 22:13:23.486551 2583 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 13 22:13:25.138008 ip-10-0-133-179 kubenswrapper[2583]: W0713 22:13:23.486553 2583 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 13 22:13:25.138008 ip-10-0-133-179 kubenswrapper[2583]: W0713 22:13:23.486556 2583 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 13 22:13:25.138008 ip-10-0-133-179 kubenswrapper[2583]: W0713 22:13:23.486558 2583 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 13 22:13:25.138008 ip-10-0-133-179 kubenswrapper[2583]: W0713 22:13:23.486561 2583 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 13 22:13:25.138008 ip-10-0-133-179 kubenswrapper[2583]: W0713 22:13:23.486563 2583 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 13 22:13:25.138008 ip-10-0-133-179 kubenswrapper[2583]: W0713 22:13:23.486566 2583 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 13 22:13:25.138008 ip-10-0-133-179 kubenswrapper[2583]: W0713 22:13:23.486568 2583 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 13 22:13:25.138008 ip-10-0-133-179 kubenswrapper[2583]: W0713 22:13:23.486571 2583 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 13 22:13:25.138008 ip-10-0-133-179 kubenswrapper[2583]: W0713 22:13:23.486573 2583 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 13 22:13:25.139085 ip-10-0-133-179 kubenswrapper[2583]: W0713 22:13:23.486576 2583 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 13 22:13:25.139085 ip-10-0-133-179 kubenswrapper[2583]: W0713 22:13:23.486579 2583 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 13 22:13:25.139085 ip-10-0-133-179 kubenswrapper[2583]: W0713 22:13:23.486583 2583 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 13 22:13:25.139085 ip-10-0-133-179 kubenswrapper[2583]: W0713 22:13:23.486585 2583 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 13 22:13:25.139085 ip-10-0-133-179 kubenswrapper[2583]: W0713 22:13:23.486588 2583 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 13 22:13:25.139085 ip-10-0-133-179 kubenswrapper[2583]: W0713 22:13:23.486591 2583 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 13 22:13:25.139085 ip-10-0-133-179 kubenswrapper[2583]: W0713 22:13:23.486593 2583 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 13 22:13:25.139085 ip-10-0-133-179 kubenswrapper[2583]: W0713 22:13:23.486596 2583 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 13 22:13:25.139085 ip-10-0-133-179 kubenswrapper[2583]: W0713 22:13:23.486598 2583 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 13 22:13:25.139085 ip-10-0-133-179 kubenswrapper[2583]: W0713 22:13:23.486601 2583 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 13 22:13:25.139085 ip-10-0-133-179 kubenswrapper[2583]: W0713 22:13:23.486603 2583 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 13 22:13:25.139085 ip-10-0-133-179 kubenswrapper[2583]: W0713 22:13:23.486606 2583 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 13 22:13:25.139085 ip-10-0-133-179 kubenswrapper[2583]: W0713 22:13:23.486608 2583 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 13 22:13:25.139085 ip-10-0-133-179 kubenswrapper[2583]: W0713 22:13:23.486611 2583 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 13 22:13:25.139085 ip-10-0-133-179 kubenswrapper[2583]: W0713 22:13:23.487318 2583 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 13 22:13:25.139085 ip-10-0-133-179 kubenswrapper[2583]: W0713 22:13:23.487325 2583 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 13 22:13:25.139085 ip-10-0-133-179 kubenswrapper[2583]: W0713 22:13:23.487328 2583 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 13 22:13:25.139085 ip-10-0-133-179 kubenswrapper[2583]: W0713 22:13:23.487331 2583 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 13 22:13:25.139085 ip-10-0-133-179 kubenswrapper[2583]: W0713 22:13:23.487334 2583 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 13 22:13:25.139085 ip-10-0-133-179 kubenswrapper[2583]: W0713 22:13:23.487338 2583 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 13 22:13:25.139668 ip-10-0-133-179 kubenswrapper[2583]: W0713 22:13:23.487340 2583 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 13 22:13:25.139668 ip-10-0-133-179 kubenswrapper[2583]: W0713 22:13:23.487343 2583 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 13 22:13:25.139668 ip-10-0-133-179 kubenswrapper[2583]: W0713 22:13:23.487346 2583 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 13 22:13:25.139668 ip-10-0-133-179 kubenswrapper[2583]: W0713 22:13:23.487348 2583 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 13 22:13:25.139668 ip-10-0-133-179 kubenswrapper[2583]: W0713 22:13:23.487351 2583 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 13 22:13:25.139668 ip-10-0-133-179 kubenswrapper[2583]: W0713 22:13:23.487353 2583 feature_gate.go:328] unrecognized feature gate: Example2 Jul 13 22:13:25.139668 ip-10-0-133-179 kubenswrapper[2583]: W0713 22:13:23.487356 2583 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 13 22:13:25.139668 ip-10-0-133-179 kubenswrapper[2583]: W0713 22:13:23.487359 2583 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 13 22:13:25.139668 ip-10-0-133-179 kubenswrapper[2583]: W0713 22:13:23.487361 2583 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 13 22:13:25.139668 ip-10-0-133-179 kubenswrapper[2583]: W0713 22:13:23.487364 2583 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 13 22:13:25.139668 ip-10-0-133-179 kubenswrapper[2583]: W0713 22:13:23.487366 2583 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 13 22:13:25.139668 ip-10-0-133-179 kubenswrapper[2583]: W0713 22:13:23.487369 2583 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 13 22:13:25.139668 ip-10-0-133-179 kubenswrapper[2583]: W0713 22:13:23.487371 2583 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 13 22:13:25.139668 ip-10-0-133-179 kubenswrapper[2583]: W0713 22:13:23.487374 2583 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 13 22:13:25.139668 ip-10-0-133-179 kubenswrapper[2583]: W0713 22:13:23.487376 2583 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 13 22:13:25.139668 ip-10-0-133-179 kubenswrapper[2583]: W0713 22:13:23.487379 2583 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 13 22:13:25.139668 ip-10-0-133-179 kubenswrapper[2583]: W0713 22:13:23.487382 2583 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 13 22:13:25.139668 ip-10-0-133-179 kubenswrapper[2583]: W0713 22:13:23.487385 2583 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 13 22:13:25.139668 ip-10-0-133-179 kubenswrapper[2583]: W0713 22:13:23.487387 2583 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 13 22:13:25.139668 ip-10-0-133-179 kubenswrapper[2583]: W0713 22:13:23.487390 2583 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 13 22:13:25.140215 ip-10-0-133-179 kubenswrapper[2583]: W0713 22:13:23.487393 2583 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 13 22:13:25.140215 ip-10-0-133-179 kubenswrapper[2583]: W0713 22:13:23.487395 2583 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 13 22:13:25.140215 ip-10-0-133-179 kubenswrapper[2583]: W0713 22:13:23.487398 2583 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 13 22:13:25.140215 ip-10-0-133-179 kubenswrapper[2583]: W0713 22:13:23.487401 2583 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 13 22:13:25.140215 ip-10-0-133-179 kubenswrapper[2583]: W0713 22:13:23.487403 2583 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 13 22:13:25.140215 ip-10-0-133-179 kubenswrapper[2583]: W0713 22:13:23.487405 2583 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 13 22:13:25.140215 ip-10-0-133-179 kubenswrapper[2583]: W0713 22:13:23.487408 2583 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 13 22:13:25.140215 ip-10-0-133-179 kubenswrapper[2583]: W0713 22:13:23.487411 2583 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 13 22:13:25.140215 ip-10-0-133-179 kubenswrapper[2583]: W0713 22:13:23.487413 2583 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 13 22:13:25.140215 ip-10-0-133-179 kubenswrapper[2583]: W0713 22:13:23.487416 2583 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 13 22:13:25.140215 ip-10-0-133-179 kubenswrapper[2583]: W0713 22:13:23.487418 2583 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 13 22:13:25.140215 ip-10-0-133-179 kubenswrapper[2583]: W0713 22:13:23.487422 2583 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 13 22:13:25.140215 ip-10-0-133-179 kubenswrapper[2583]: W0713 22:13:23.487426 2583 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 13 22:13:25.140215 ip-10-0-133-179 kubenswrapper[2583]: W0713 22:13:23.487429 2583 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 13 22:13:25.140215 ip-10-0-133-179 kubenswrapper[2583]: W0713 22:13:23.487431 2583 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 13 22:13:25.140215 ip-10-0-133-179 kubenswrapper[2583]: W0713 22:13:23.487434 2583 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 13 22:13:25.140215 ip-10-0-133-179 kubenswrapper[2583]: W0713 22:13:23.487436 2583 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 13 22:13:25.140215 ip-10-0-133-179 kubenswrapper[2583]: W0713 22:13:23.487438 2583 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 13 22:13:25.140215 ip-10-0-133-179 kubenswrapper[2583]: W0713 22:13:23.487441 2583 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 13 22:13:25.140215 ip-10-0-133-179 kubenswrapper[2583]: W0713 22:13:23.487443 2583 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 13 22:13:25.140790 ip-10-0-133-179 kubenswrapper[2583]: W0713 22:13:23.487446 2583 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 13 22:13:25.140790 ip-10-0-133-179 kubenswrapper[2583]: W0713 22:13:23.487449 2583 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 13 22:13:25.140790 ip-10-0-133-179 kubenswrapper[2583]: W0713 22:13:23.487451 2583 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 13 22:13:25.140790 ip-10-0-133-179 kubenswrapper[2583]: W0713 22:13:23.487454 2583 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 13 22:13:25.140790 ip-10-0-133-179 kubenswrapper[2583]: W0713 22:13:23.487456 2583 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 13 22:13:25.140790 ip-10-0-133-179 kubenswrapper[2583]: W0713 22:13:23.487458 2583 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 13 22:13:25.140790 ip-10-0-133-179 kubenswrapper[2583]: W0713 22:13:23.487462 2583 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 13 22:13:25.140790 ip-10-0-133-179 kubenswrapper[2583]: W0713 22:13:23.487466 2583 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 13 22:13:25.140790 ip-10-0-133-179 kubenswrapper[2583]: W0713 22:13:23.487470 2583 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 13 22:13:25.140790 ip-10-0-133-179 kubenswrapper[2583]: W0713 22:13:23.487473 2583 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 13 22:13:25.140790 ip-10-0-133-179 kubenswrapper[2583]: W0713 22:13:23.487475 2583 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 13 22:13:25.140790 ip-10-0-133-179 kubenswrapper[2583]: W0713 22:13:23.487478 2583 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 13 22:13:25.140790 ip-10-0-133-179 kubenswrapper[2583]: W0713 22:13:23.487481 2583 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 13 22:13:25.140790 ip-10-0-133-179 kubenswrapper[2583]: W0713 22:13:23.487484 2583 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 13 22:13:25.140790 ip-10-0-133-179 kubenswrapper[2583]: W0713 22:13:23.487487 2583 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 13 22:13:25.140790 ip-10-0-133-179 kubenswrapper[2583]: W0713 22:13:23.487489 2583 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 13 22:13:25.140790 ip-10-0-133-179 kubenswrapper[2583]: W0713 22:13:23.487492 2583 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 13 22:13:25.140790 ip-10-0-133-179 kubenswrapper[2583]: W0713 22:13:23.487495 2583 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 13 22:13:25.140790 ip-10-0-133-179 kubenswrapper[2583]: W0713 22:13:23.487497 2583 feature_gate.go:328] unrecognized feature gate: Example Jul 13 22:13:25.141370 ip-10-0-133-179 kubenswrapper[2583]: W0713 22:13:23.487500 2583 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 13 22:13:25.141370 ip-10-0-133-179 kubenswrapper[2583]: W0713 22:13:23.487503 2583 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 13 22:13:25.141370 ip-10-0-133-179 kubenswrapper[2583]: W0713 22:13:23.487505 2583 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 13 22:13:25.141370 ip-10-0-133-179 kubenswrapper[2583]: W0713 22:13:23.487509 2583 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 13 22:13:25.141370 ip-10-0-133-179 kubenswrapper[2583]: W0713 22:13:23.487512 2583 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 13 22:13:25.141370 ip-10-0-133-179 kubenswrapper[2583]: W0713 22:13:23.487515 2583 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 13 22:13:25.141370 ip-10-0-133-179 kubenswrapper[2583]: W0713 22:13:23.487520 2583 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 13 22:13:25.141370 ip-10-0-133-179 kubenswrapper[2583]: W0713 22:13:23.487523 2583 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 13 22:13:25.141370 ip-10-0-133-179 kubenswrapper[2583]: W0713 22:13:23.487526 2583 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 13 22:13:25.141370 ip-10-0-133-179 kubenswrapper[2583]: W0713 22:13:23.487529 2583 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 13 22:13:25.141370 ip-10-0-133-179 kubenswrapper[2583]: W0713 22:13:23.487532 2583 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 13 22:13:25.141370 ip-10-0-133-179 kubenswrapper[2583]: W0713 22:13:23.487535 2583 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 13 22:13:25.141370 ip-10-0-133-179 kubenswrapper[2583]: W0713 22:13:23.487538 2583 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 13 22:13:25.141370 ip-10-0-133-179 kubenswrapper[2583]: W0713 22:13:23.487541 2583 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 13 22:13:25.141370 ip-10-0-133-179 kubenswrapper[2583]: W0713 22:13:23.487543 2583 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 13 22:13:25.141370 ip-10-0-133-179 kubenswrapper[2583]: W0713 22:13:23.487547 2583 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 13 22:13:25.141370 ip-10-0-133-179 kubenswrapper[2583]: W0713 22:13:23.487550 2583 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 13 22:13:25.141370 ip-10-0-133-179 kubenswrapper[2583]: W0713 22:13:23.487552 2583 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 13 22:13:25.141370 ip-10-0-133-179 kubenswrapper[2583]: W0713 22:13:23.487555 2583 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 13 22:13:25.141370 ip-10-0-133-179 kubenswrapper[2583]: W0713 22:13:23.487557 2583 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 13 22:13:25.141951 ip-10-0-133-179 kubenswrapper[2583]: W0713 22:13:23.487560 2583 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 13 22:13:25.141951 ip-10-0-133-179 kubenswrapper[2583]: W0713 22:13:23.487564 2583 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 13 22:13:25.141951 ip-10-0-133-179 kubenswrapper[2583]: W0713 22:13:23.487566 2583 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 13 22:13:25.141951 ip-10-0-133-179 kubenswrapper[2583]: W0713 22:13:23.487569 2583 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 13 22:13:25.141951 ip-10-0-133-179 kubenswrapper[2583]: W0713 22:13:23.487572 2583 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 13 22:13:25.141951 ip-10-0-133-179 kubenswrapper[2583]: W0713 22:13:23.487574 2583 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 13 22:13:25.141951 ip-10-0-133-179 kubenswrapper[2583]: W0713 22:13:23.487577 2583 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 13 22:13:25.141951 ip-10-0-133-179 kubenswrapper[2583]: W0713 22:13:23.487579 2583 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 13 22:13:25.141951 ip-10-0-133-179 kubenswrapper[2583]: W0713 22:13:23.487582 2583 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 13 22:13:25.141951 ip-10-0-133-179 kubenswrapper[2583]: W0713 22:13:23.487584 2583 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 13 22:13:25.141951 ip-10-0-133-179 kubenswrapper[2583]: W0713 22:13:23.487586 2583 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 13 22:13:25.141951 ip-10-0-133-179 kubenswrapper[2583]: W0713 22:13:23.487589 2583 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 13 22:13:25.141951 ip-10-0-133-179 kubenswrapper[2583]: W0713 22:13:23.487591 2583 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 13 22:13:25.141951 ip-10-0-133-179 kubenswrapper[2583]: W0713 22:13:23.487594 2583 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 13 22:13:25.141951 ip-10-0-133-179 kubenswrapper[2583]: W0713 22:13:23.487991 2583 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 13 22:13:25.141951 ip-10-0-133-179 kubenswrapper[2583]: W0713 22:13:23.487997 2583 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 13 22:13:25.141951 ip-10-0-133-179 kubenswrapper[2583]: W0713 22:13:23.488000 2583 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 13 22:13:25.141951 ip-10-0-133-179 kubenswrapper[2583]: W0713 22:13:23.488003 2583 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 13 22:13:25.141951 ip-10-0-133-179 kubenswrapper[2583]: W0713 22:13:23.488007 2583 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 13 22:13:25.141951 ip-10-0-133-179 kubenswrapper[2583]: W0713 22:13:23.488009 2583 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 13 22:13:25.142518 ip-10-0-133-179 kubenswrapper[2583]: W0713 22:13:23.488012 2583 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 13 22:13:25.142518 ip-10-0-133-179 kubenswrapper[2583]: W0713 22:13:23.488015 2583 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 13 22:13:25.142518 ip-10-0-133-179 kubenswrapper[2583]: W0713 22:13:23.488017 2583 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 13 22:13:25.142518 ip-10-0-133-179 kubenswrapper[2583]: W0713 22:13:23.488020 2583 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 13 22:13:25.142518 ip-10-0-133-179 kubenswrapper[2583]: W0713 22:13:23.488023 2583 feature_gate.go:328] unrecognized feature gate: Example2 Jul 13 22:13:25.142518 ip-10-0-133-179 kubenswrapper[2583]: W0713 22:13:23.488025 2583 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 13 22:13:25.142518 ip-10-0-133-179 kubenswrapper[2583]: W0713 22:13:23.488028 2583 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 13 22:13:25.142518 ip-10-0-133-179 kubenswrapper[2583]: W0713 22:13:23.488031 2583 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 13 22:13:25.142518 ip-10-0-133-179 kubenswrapper[2583]: W0713 22:13:23.488033 2583 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 13 22:13:25.142518 ip-10-0-133-179 kubenswrapper[2583]: W0713 22:13:23.488036 2583 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 13 22:13:25.142518 ip-10-0-133-179 kubenswrapper[2583]: W0713 22:13:23.488038 2583 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 13 22:13:25.142518 ip-10-0-133-179 kubenswrapper[2583]: W0713 22:13:23.488041 2583 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 13 22:13:25.142518 ip-10-0-133-179 kubenswrapper[2583]: W0713 22:13:23.488044 2583 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 13 22:13:25.142518 ip-10-0-133-179 kubenswrapper[2583]: W0713 22:13:23.488049 2583 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 13 22:13:25.142518 ip-10-0-133-179 kubenswrapper[2583]: W0713 22:13:23.488052 2583 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 13 22:13:25.142518 ip-10-0-133-179 kubenswrapper[2583]: W0713 22:13:23.488055 2583 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 13 22:13:25.142518 ip-10-0-133-179 kubenswrapper[2583]: W0713 22:13:23.488057 2583 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 13 22:13:25.142518 ip-10-0-133-179 kubenswrapper[2583]: W0713 22:13:23.488062 2583 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 13 22:13:25.142518 ip-10-0-133-179 kubenswrapper[2583]: W0713 22:13:23.488065 2583 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 13 22:13:25.143109 ip-10-0-133-179 kubenswrapper[2583]: W0713 22:13:23.488068 2583 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 13 22:13:25.143109 ip-10-0-133-179 kubenswrapper[2583]: W0713 22:13:23.488071 2583 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 13 22:13:25.143109 ip-10-0-133-179 kubenswrapper[2583]: W0713 22:13:23.488073 2583 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 13 22:13:25.143109 ip-10-0-133-179 kubenswrapper[2583]: W0713 22:13:23.488076 2583 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 13 22:13:25.143109 ip-10-0-133-179 kubenswrapper[2583]: W0713 22:13:23.488078 2583 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 13 22:13:25.143109 ip-10-0-133-179 kubenswrapper[2583]: W0713 22:13:23.488081 2583 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 13 22:13:25.143109 ip-10-0-133-179 kubenswrapper[2583]: W0713 22:13:23.488084 2583 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 13 22:13:25.143109 ip-10-0-133-179 kubenswrapper[2583]: W0713 22:13:23.488087 2583 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 13 22:13:25.143109 ip-10-0-133-179 kubenswrapper[2583]: W0713 22:13:23.488090 2583 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 13 22:13:25.143109 ip-10-0-133-179 kubenswrapper[2583]: W0713 22:13:23.488093 2583 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 13 22:13:25.143109 ip-10-0-133-179 kubenswrapper[2583]: W0713 22:13:23.488095 2583 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 13 22:13:25.143109 ip-10-0-133-179 kubenswrapper[2583]: W0713 22:13:23.488097 2583 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 13 22:13:25.143109 ip-10-0-133-179 kubenswrapper[2583]: W0713 22:13:23.488100 2583 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 13 22:13:25.143109 ip-10-0-133-179 kubenswrapper[2583]: W0713 22:13:23.488104 2583 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 13 22:13:25.143109 ip-10-0-133-179 kubenswrapper[2583]: W0713 22:13:23.488106 2583 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 13 22:13:25.143109 ip-10-0-133-179 kubenswrapper[2583]: W0713 22:13:23.488109 2583 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 13 22:13:25.143109 ip-10-0-133-179 kubenswrapper[2583]: W0713 22:13:23.488111 2583 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 13 22:13:25.143109 ip-10-0-133-179 kubenswrapper[2583]: W0713 22:13:23.488113 2583 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 13 22:13:25.143109 ip-10-0-133-179 kubenswrapper[2583]: W0713 22:13:23.488116 2583 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 13 22:13:25.143109 ip-10-0-133-179 kubenswrapper[2583]: W0713 22:13:23.488118 2583 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 13 22:13:25.143662 ip-10-0-133-179 kubenswrapper[2583]: W0713 22:13:23.488121 2583 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 13 22:13:25.143662 ip-10-0-133-179 kubenswrapper[2583]: W0713 22:13:23.488123 2583 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 13 22:13:25.143662 ip-10-0-133-179 kubenswrapper[2583]: W0713 22:13:23.488126 2583 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 13 22:13:25.143662 ip-10-0-133-179 kubenswrapper[2583]: W0713 22:13:23.488128 2583 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 13 22:13:25.143662 ip-10-0-133-179 kubenswrapper[2583]: W0713 22:13:23.488131 2583 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 13 22:13:25.143662 ip-10-0-133-179 kubenswrapper[2583]: W0713 22:13:23.488133 2583 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 13 22:13:25.143662 ip-10-0-133-179 kubenswrapper[2583]: W0713 22:13:23.488137 2583 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 13 22:13:25.143662 ip-10-0-133-179 kubenswrapper[2583]: W0713 22:13:23.488140 2583 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 13 22:13:25.143662 ip-10-0-133-179 kubenswrapper[2583]: W0713 22:13:23.488142 2583 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 13 22:13:25.143662 ip-10-0-133-179 kubenswrapper[2583]: W0713 22:13:23.488145 2583 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 13 22:13:25.143662 ip-10-0-133-179 kubenswrapper[2583]: W0713 22:13:23.488147 2583 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 13 22:13:25.143662 ip-10-0-133-179 kubenswrapper[2583]: W0713 22:13:23.488150 2583 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 13 22:13:25.143662 ip-10-0-133-179 kubenswrapper[2583]: W0713 22:13:23.488152 2583 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 13 22:13:25.143662 ip-10-0-133-179 kubenswrapper[2583]: W0713 22:13:23.488155 2583 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 13 22:13:25.143662 ip-10-0-133-179 kubenswrapper[2583]: W0713 22:13:23.488158 2583 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 13 22:13:25.143662 ip-10-0-133-179 kubenswrapper[2583]: W0713 22:13:23.488160 2583 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 13 22:13:25.143662 ip-10-0-133-179 kubenswrapper[2583]: W0713 22:13:23.488163 2583 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 13 22:13:25.143662 ip-10-0-133-179 kubenswrapper[2583]: W0713 22:13:23.488165 2583 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 13 22:13:25.143662 ip-10-0-133-179 kubenswrapper[2583]: W0713 22:13:23.488168 2583 feature_gate.go:328] unrecognized feature gate: Example Jul 13 22:13:25.143662 ip-10-0-133-179 kubenswrapper[2583]: W0713 22:13:23.488170 2583 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 13 22:13:25.144226 ip-10-0-133-179 kubenswrapper[2583]: W0713 22:13:23.488173 2583 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 13 22:13:25.144226 ip-10-0-133-179 kubenswrapper[2583]: W0713 22:13:23.488175 2583 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 13 22:13:25.144226 ip-10-0-133-179 kubenswrapper[2583]: W0713 22:13:23.488178 2583 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 13 22:13:25.144226 ip-10-0-133-179 kubenswrapper[2583]: W0713 22:13:23.488180 2583 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 13 22:13:25.144226 ip-10-0-133-179 kubenswrapper[2583]: W0713 22:13:23.488183 2583 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 13 22:13:25.144226 ip-10-0-133-179 kubenswrapper[2583]: W0713 22:13:23.488186 2583 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 13 22:13:25.144226 ip-10-0-133-179 kubenswrapper[2583]: W0713 22:13:23.488188 2583 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 13 22:13:25.144226 ip-10-0-133-179 kubenswrapper[2583]: W0713 22:13:23.488191 2583 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 13 22:13:25.144226 ip-10-0-133-179 kubenswrapper[2583]: W0713 22:13:23.488193 2583 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 13 22:13:25.144226 ip-10-0-133-179 kubenswrapper[2583]: W0713 22:13:23.488196 2583 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 13 22:13:25.144226 ip-10-0-133-179 kubenswrapper[2583]: W0713 22:13:23.488198 2583 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 13 22:13:25.144226 ip-10-0-133-179 kubenswrapper[2583]: W0713 22:13:23.488201 2583 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 13 22:13:25.144226 ip-10-0-133-179 kubenswrapper[2583]: W0713 22:13:23.488203 2583 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 13 22:13:25.144226 ip-10-0-133-179 kubenswrapper[2583]: W0713 22:13:23.488206 2583 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 13 22:13:25.144226 ip-10-0-133-179 kubenswrapper[2583]: W0713 22:13:23.488208 2583 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 13 22:13:25.144226 ip-10-0-133-179 kubenswrapper[2583]: W0713 22:13:23.488213 2583 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 13 22:13:25.144226 ip-10-0-133-179 kubenswrapper[2583]: W0713 22:13:23.488215 2583 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 13 22:13:25.144226 ip-10-0-133-179 kubenswrapper[2583]: W0713 22:13:23.488218 2583 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 13 22:13:25.144226 ip-10-0-133-179 kubenswrapper[2583]: W0713 22:13:23.488237 2583 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 13 22:13:25.144778 ip-10-0-133-179 kubenswrapper[2583]: W0713 22:13:23.488241 2583 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 13 22:13:25.144778 ip-10-0-133-179 kubenswrapper[2583]: W0713 22:13:23.488244 2583 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 13 22:13:25.144778 ip-10-0-133-179 kubenswrapper[2583]: W0713 22:13:23.488247 2583 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 13 22:13:25.144778 ip-10-0-133-179 kubenswrapper[2583]: W0713 22:13:23.488250 2583 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 13 22:13:25.144778 ip-10-0-133-179 kubenswrapper[2583]: W0713 22:13:23.488252 2583 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 13 22:13:25.144778 ip-10-0-133-179 kubenswrapper[2583]: W0713 22:13:23.488255 2583 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 13 22:13:25.144778 ip-10-0-133-179 kubenswrapper[2583]: W0713 22:13:23.488257 2583 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 13 22:13:25.144778 ip-10-0-133-179 kubenswrapper[2583]: W0713 22:13:23.488260 2583 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 13 22:13:25.144778 ip-10-0-133-179 kubenswrapper[2583]: W0713 22:13:23.488262 2583 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 13 22:13:25.144778 ip-10-0-133-179 kubenswrapper[2583]: W0713 22:13:23.488265 2583 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 13 22:13:25.144778 ip-10-0-133-179 kubenswrapper[2583]: W0713 22:13:23.488269 2583 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 13 22:13:25.144778 ip-10-0-133-179 kubenswrapper[2583]: W0713 22:13:23.488272 2583 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 13 22:13:25.144778 ip-10-0-133-179 kubenswrapper[2583]: W0713 22:13:23.488275 2583 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 13 22:13:25.144778 ip-10-0-133-179 kubenswrapper[2583]: W0713 22:13:23.488278 2583 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 13 22:13:25.144778 ip-10-0-133-179 kubenswrapper[2583]: W0713 22:13:23.488281 2583 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 13 22:13:25.144778 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:13:23.488354 2583 flags.go:64] FLAG: --address="0.0.0.0" Jul 13 22:13:25.144778 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:13:23.488364 2583 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]" Jul 13 22:13:25.144778 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:13:23.488373 2583 flags.go:64] FLAG: --anonymous-auth="true" Jul 13 22:13:25.144778 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:13:23.488380 2583 flags.go:64] FLAG: --application-metrics-count-limit="100" Jul 13 22:13:25.144778 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:13:23.488387 2583 flags.go:64] FLAG: --authentication-token-webhook="false" Jul 13 22:13:25.145333 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:13:23.488390 2583 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s" Jul 13 22:13:25.145333 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:13:23.488394 2583 flags.go:64] FLAG: --authorization-mode="AlwaysAllow" Jul 13 22:13:25.145333 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:13:23.488398 2583 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s" Jul 13 22:13:25.145333 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:13:23.488402 2583 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s" Jul 13 22:13:25.145333 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:13:23.488404 2583 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id" Jul 13 22:13:25.145333 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:13:23.488408 2583 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig" Jul 13 22:13:25.145333 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:13:23.488411 2583 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki" Jul 13 22:13:25.145333 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:13:23.488414 2583 flags.go:64] FLAG: --cgroup-driver="cgroupfs" Jul 13 22:13:25.145333 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:13:23.488419 2583 flags.go:64] FLAG: --cgroup-root="" Jul 13 22:13:25.145333 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:13:23.488422 2583 flags.go:64] FLAG: --cgroups-per-qos="true" Jul 13 22:13:25.145333 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:13:23.488425 2583 flags.go:64] FLAG: --client-ca-file="" Jul 13 22:13:25.145333 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:13:23.488429 2583 flags.go:64] FLAG: --cloud-provider="external" Jul 13 22:13:25.145333 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:13:23.488432 2583 flags.go:64] FLAG: --cluster-dns="[]" Jul 13 22:13:25.145333 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:13:23.488436 2583 flags.go:64] FLAG: --cluster-domain="" Jul 13 22:13:25.145333 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:13:23.488439 2583 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf" Jul 13 22:13:25.145333 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:13:23.488442 2583 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d" Jul 13 22:13:25.145333 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:13:23.488445 2583 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json" Jul 13 22:13:25.145333 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:13:23.488449 2583 flags.go:64] FLAG: --container-log-max-files="5" Jul 13 22:13:25.145333 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:13:23.488453 2583 flags.go:64] FLAG: --container-log-max-size="10Mi" Jul 13 22:13:25.145333 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:13:23.488456 2583 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock" Jul 13 22:13:25.145333 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:13:23.488459 2583 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock" Jul 13 22:13:25.145333 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:13:23.488462 2583 flags.go:64] FLAG: --containerd-namespace="k8s.io" Jul 13 22:13:25.145333 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:13:23.488465 2583 flags.go:64] FLAG: --contention-profiling="false" Jul 13 22:13:25.145333 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:13:23.488468 2583 flags.go:64] FLAG: --cpu-cfs-quota="true" Jul 13 22:13:25.146143 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:13:23.488471 2583 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms" Jul 13 22:13:25.146143 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:13:23.488475 2583 flags.go:64] FLAG: --cpu-manager-policy="none" Jul 13 22:13:25.146143 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:13:23.488478 2583 flags.go:64] FLAG: --cpu-manager-policy-options="" Jul 13 22:13:25.146143 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:13:23.488482 2583 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s" Jul 13 22:13:25.146143 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:13:23.488484 2583 flags.go:64] FLAG: --enable-controller-attach-detach="true" Jul 13 22:13:25.146143 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:13:23.488487 2583 flags.go:64] FLAG: --enable-debugging-handlers="true" Jul 13 22:13:25.146143 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:13:23.488490 2583 flags.go:64] FLAG: --enable-load-reader="false" Jul 13 22:13:25.146143 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:13:23.488493 2583 flags.go:64] FLAG: --enable-server="true" Jul 13 22:13:25.146143 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:13:23.488497 2583 flags.go:64] FLAG: --enforce-node-allocatable="[pods]" Jul 13 22:13:25.146143 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:13:23.488501 2583 flags.go:64] FLAG: --event-burst="100" Jul 13 22:13:25.146143 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:13:23.488504 2583 flags.go:64] FLAG: --event-qps="50" Jul 13 22:13:25.146143 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:13:23.488507 2583 flags.go:64] FLAG: --event-storage-age-limit="default=0" Jul 13 22:13:25.146143 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:13:23.488510 2583 flags.go:64] FLAG: --event-storage-event-limit="default=0" Jul 13 22:13:25.146143 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:13:23.488513 2583 flags.go:64] FLAG: --eviction-hard="" Jul 13 22:13:25.146143 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:13:23.488517 2583 flags.go:64] FLAG: --eviction-max-pod-grace-period="0" Jul 13 22:13:25.146143 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:13:23.488520 2583 flags.go:64] FLAG: --eviction-minimum-reclaim="" Jul 13 22:13:25.146143 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:13:23.488524 2583 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s" Jul 13 22:13:25.146143 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:13:23.488527 2583 flags.go:64] FLAG: --eviction-soft="" Jul 13 22:13:25.146143 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:13:23.488530 2583 flags.go:64] FLAG: --eviction-soft-grace-period="" Jul 13 22:13:25.146143 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:13:23.488534 2583 flags.go:64] FLAG: --exit-on-lock-contention="false" Jul 13 22:13:25.146143 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:13:23.488538 2583 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false" Jul 13 22:13:25.146143 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:13:23.488541 2583 flags.go:64] FLAG: --experimental-mounter-path="" Jul 13 22:13:25.146143 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:13:23.488543 2583 flags.go:64] FLAG: --fail-cgroupv1="false" Jul 13 22:13:25.146143 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:13:23.488546 2583 flags.go:64] FLAG: --fail-swap-on="true" Jul 13 22:13:25.146143 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:13:23.488549 2583 flags.go:64] FLAG: --feature-gates="" Jul 13 22:13:25.146902 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:13:23.488553 2583 flags.go:64] FLAG: --file-check-frequency="20s" Jul 13 22:13:25.146902 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:13:23.488556 2583 flags.go:64] FLAG: --global-housekeeping-interval="1m0s" Jul 13 22:13:25.146902 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:13:23.488559 2583 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge" Jul 13 22:13:25.146902 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:13:23.488562 2583 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1" Jul 13 22:13:25.146902 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:13:23.488565 2583 flags.go:64] FLAG: --healthz-port="10248" Jul 13 22:13:25.146902 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:13:23.488568 2583 flags.go:64] FLAG: --help="false" Jul 13 22:13:25.146902 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:13:23.488571 2583 flags.go:64] FLAG: --hostname-override="ip-10-0-133-179.ec2.internal" Jul 13 22:13:25.146902 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:13:23.488574 2583 flags.go:64] FLAG: --housekeeping-interval="10s" Jul 13 22:13:25.146902 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:13:23.488577 2583 flags.go:64] FLAG: --http-check-frequency="20s" Jul 13 22:13:25.146902 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:13:23.488580 2583 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins" Jul 13 22:13:25.146902 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:13:23.488584 2583 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml" Jul 13 22:13:25.146902 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:13:23.488587 2583 flags.go:64] FLAG: --image-gc-high-threshold="85" Jul 13 22:13:25.146902 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:13:23.488590 2583 flags.go:64] FLAG: --image-gc-low-threshold="80" Jul 13 22:13:25.146902 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:13:23.488593 2583 flags.go:64] FLAG: --image-service-endpoint="" Jul 13 22:13:25.146902 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:13:23.488596 2583 flags.go:64] FLAG: --kernel-memcg-notification="false" Jul 13 22:13:25.146902 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:13:23.488599 2583 flags.go:64] FLAG: --kube-api-burst="100" Jul 13 22:13:25.146902 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:13:23.488602 2583 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf" Jul 13 22:13:25.146902 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:13:23.488605 2583 flags.go:64] FLAG: --kube-api-qps="50" Jul 13 22:13:25.146902 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:13:23.488608 2583 flags.go:64] FLAG: --kube-reserved="" Jul 13 22:13:25.146902 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:13:23.488611 2583 flags.go:64] FLAG: --kube-reserved-cgroup="" Jul 13 22:13:25.146902 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:13:23.488614 2583 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig" Jul 13 22:13:25.146902 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:13:23.488617 2583 flags.go:64] FLAG: --kubelet-cgroups="" Jul 13 22:13:25.146902 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:13:23.488619 2583 flags.go:64] FLAG: --local-storage-capacity-isolation="true" Jul 13 22:13:25.146902 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:13:23.488624 2583 flags.go:64] FLAG: --lock-file="" Jul 13 22:13:25.148135 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:13:23.488627 2583 flags.go:64] FLAG: --log-cadvisor-usage="false" Jul 13 22:13:25.148135 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:13:23.488630 2583 flags.go:64] FLAG: --log-flush-frequency="5s" Jul 13 22:13:25.148135 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:13:23.488634 2583 flags.go:64] FLAG: --log-json-info-buffer-size="0" Jul 13 22:13:25.148135 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:13:23.488640 2583 flags.go:64] FLAG: --log-json-split-stream="false" Jul 13 22:13:25.148135 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:13:23.488643 2583 flags.go:64] FLAG: --log-text-info-buffer-size="0" Jul 13 22:13:25.148135 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:13:23.488646 2583 flags.go:64] FLAG: --log-text-split-stream="false" Jul 13 22:13:25.148135 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:13:23.488649 2583 flags.go:64] FLAG: --logging-format="text" Jul 13 22:13:25.148135 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:13:23.488651 2583 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id" Jul 13 22:13:25.148135 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:13:23.488655 2583 flags.go:64] FLAG: --make-iptables-util-chains="true" Jul 13 22:13:25.148135 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:13:23.488658 2583 flags.go:64] FLAG: --manifest-url="" Jul 13 22:13:25.148135 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:13:23.488660 2583 flags.go:64] FLAG: --manifest-url-header="" Jul 13 22:13:25.148135 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:13:23.488665 2583 flags.go:64] FLAG: --max-housekeeping-interval="15s" Jul 13 22:13:25.148135 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:13:23.488668 2583 flags.go:64] FLAG: --max-open-files="1000000" Jul 13 22:13:25.148135 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:13:23.488672 2583 flags.go:64] FLAG: --max-pods="110" Jul 13 22:13:25.148135 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:13:23.488674 2583 flags.go:64] FLAG: --maximum-dead-containers="-1" Jul 13 22:13:25.148135 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:13:23.488677 2583 flags.go:64] FLAG: --maximum-dead-containers-per-container="1" Jul 13 22:13:25.148135 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:13:23.488680 2583 flags.go:64] FLAG: --memory-manager-policy="None" Jul 13 22:13:25.148135 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:13:23.488683 2583 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s" Jul 13 22:13:25.148135 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:13:23.488686 2583 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s" Jul 13 22:13:25.148135 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:13:23.488689 2583 flags.go:64] FLAG: --node-ip="0.0.0.0" Jul 13 22:13:25.148135 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:13:23.488692 2583 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel" Jul 13 22:13:25.148135 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:13:23.488699 2583 flags.go:64] FLAG: --node-status-max-images="50" Jul 13 22:13:25.148135 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:13:23.488702 2583 flags.go:64] FLAG: --node-status-update-frequency="10s" Jul 13 22:13:25.148135 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:13:23.488705 2583 flags.go:64] FLAG: --oom-score-adj="-999" Jul 13 22:13:25.148135 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:13:23.488708 2583 flags.go:64] FLAG: --pod-cidr="" Jul 13 22:13:25.148997 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:13:23.488711 2583 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:b04c0b1f73af4d021dd8f0b51bfeb302c24482c093de5f2c43654ab56c506926" Jul 13 22:13:25.148997 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:13:23.488717 2583 flags.go:64] FLAG: --pod-manifest-path="" Jul 13 22:13:25.148997 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:13:23.488720 2583 flags.go:64] FLAG: --pod-max-pids="-1" Jul 13 22:13:25.148997 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:13:23.488723 2583 flags.go:64] FLAG: --pods-per-core="0" Jul 13 22:13:25.148997 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:13:23.488726 2583 flags.go:64] FLAG: --port="10250" Jul 13 22:13:25.148997 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:13:23.488729 2583 flags.go:64] FLAG: --protect-kernel-defaults="false" Jul 13 22:13:25.148997 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:13:23.488734 2583 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-0da62c0120ab06779" Jul 13 22:13:25.148997 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:13:23.488737 2583 flags.go:64] FLAG: --qos-reserved="" Jul 13 22:13:25.148997 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:13:23.488740 2583 flags.go:64] FLAG: --read-only-port="10255" Jul 13 22:13:25.148997 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:13:23.488742 2583 flags.go:64] FLAG: --register-node="true" Jul 13 22:13:25.148997 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:13:23.488746 2583 flags.go:64] FLAG: --register-with-taints="" Jul 13 22:13:25.148997 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:13:23.488750 2583 flags.go:64] FLAG: --registry-burst="10" Jul 13 22:13:25.148997 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:13:23.488753 2583 flags.go:64] FLAG: --registry-qps="5" Jul 13 22:13:25.148997 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:13:23.488755 2583 flags.go:64] FLAG: --reserved-cpus="" Jul 13 22:13:25.148997 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:13:23.488758 2583 flags.go:64] FLAG: --reserved-memory="" Jul 13 22:13:25.148997 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:13:23.488762 2583 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf" Jul 13 22:13:25.148997 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:13:23.488765 2583 flags.go:64] FLAG: --root-dir="/var/lib/kubelet" Jul 13 22:13:25.148997 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:13:23.488768 2583 flags.go:64] FLAG: --rotate-certificates="false" Jul 13 22:13:25.148997 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:13:23.488771 2583 flags.go:64] FLAG: --rotate-server-certificates="false" Jul 13 22:13:25.148997 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:13:23.488775 2583 flags.go:64] FLAG: --runonce="false" Jul 13 22:13:25.148997 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:13:23.488777 2583 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service" Jul 13 22:13:25.148997 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:13:23.488781 2583 flags.go:64] FLAG: --runtime-request-timeout="2m0s" Jul 13 22:13:25.148997 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:13:23.488783 2583 flags.go:64] FLAG: --seccomp-default="false" Jul 13 22:13:25.148997 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:13:23.488786 2583 flags.go:64] FLAG: --serialize-image-pulls="true" Jul 13 22:13:25.148997 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:13:23.488789 2583 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s" Jul 13 22:13:25.250164 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:13:23.488792 2583 flags.go:64] FLAG: --storage-driver-db="cadvisor" Jul 13 22:13:25.250164 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:13:23.488795 2583 flags.go:64] FLAG: --storage-driver-host="localhost:8086" Jul 13 22:13:25.250164 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:13:23.488799 2583 flags.go:64] FLAG: --storage-driver-password="root" Jul 13 22:13:25.250164 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:13:23.488801 2583 flags.go:64] FLAG: --storage-driver-secure="false" Jul 13 22:13:25.250164 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:13:23.488804 2583 flags.go:64] FLAG: --storage-driver-table="stats" Jul 13 22:13:25.250164 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:13:23.488807 2583 flags.go:64] FLAG: --storage-driver-user="root" Jul 13 22:13:25.250164 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:13:23.488810 2583 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s" Jul 13 22:13:25.250164 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:13:23.488813 2583 flags.go:64] FLAG: --sync-frequency="1m0s" Jul 13 22:13:25.250164 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:13:23.488816 2583 flags.go:64] FLAG: --system-cgroups="" Jul 13 22:13:25.250164 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:13:23.488819 2583 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi" Jul 13 22:13:25.250164 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:13:23.488824 2583 flags.go:64] FLAG: --system-reserved-cgroup="" Jul 13 22:13:25.250164 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:13:23.488827 2583 flags.go:64] FLAG: --tls-cert-file="" Jul 13 22:13:25.250164 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:13:23.488830 2583 flags.go:64] FLAG: --tls-cipher-suites="[]" Jul 13 22:13:25.250164 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:13:23.488835 2583 flags.go:64] FLAG: --tls-min-version="" Jul 13 22:13:25.250164 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:13:23.488838 2583 flags.go:64] FLAG: --tls-private-key-file="" Jul 13 22:13:25.250164 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:13:23.488841 2583 flags.go:64] FLAG: --topology-manager-policy="none" Jul 13 22:13:25.250164 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:13:23.488844 2583 flags.go:64] FLAG: --topology-manager-policy-options="" Jul 13 22:13:25.250164 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:13:23.488847 2583 flags.go:64] FLAG: --topology-manager-scope="container" Jul 13 22:13:25.250164 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:13:23.488850 2583 flags.go:64] FLAG: --v="2" Jul 13 22:13:25.250164 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:13:23.488854 2583 flags.go:64] FLAG: --version="false" Jul 13 22:13:25.250164 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:13:23.488858 2583 flags.go:64] FLAG: --vmodule="" Jul 13 22:13:25.250164 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:13:23.488862 2583 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec" Jul 13 22:13:25.250164 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:13:23.488865 2583 flags.go:64] FLAG: --volume-stats-agg-period="1m0s" Jul 13 22:13:25.250164 ip-10-0-133-179 kubenswrapper[2583]: W0713 22:13:23.488978 2583 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 13 22:13:25.250164 ip-10-0-133-179 kubenswrapper[2583]: W0713 22:13:23.488982 2583 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 13 22:13:25.169731 ip-10-0-133-179 systemd[1]: Started Kubernetes Kubelet. Jul 13 22:13:25.251276 ip-10-0-133-179 kubenswrapper[2583]: W0713 22:13:23.488985 2583 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 13 22:13:25.251276 ip-10-0-133-179 kubenswrapper[2583]: W0713 22:13:23.488988 2583 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 13 22:13:25.251276 ip-10-0-133-179 kubenswrapper[2583]: W0713 22:13:23.488991 2583 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 13 22:13:25.251276 ip-10-0-133-179 kubenswrapper[2583]: W0713 22:13:23.488994 2583 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 13 22:13:25.251276 ip-10-0-133-179 kubenswrapper[2583]: W0713 22:13:23.488996 2583 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 13 22:13:25.251276 ip-10-0-133-179 kubenswrapper[2583]: W0713 22:13:23.488999 2583 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 13 22:13:25.251276 ip-10-0-133-179 kubenswrapper[2583]: W0713 22:13:23.489001 2583 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 13 22:13:25.251276 ip-10-0-133-179 kubenswrapper[2583]: W0713 22:13:23.489004 2583 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 13 22:13:25.251276 ip-10-0-133-179 kubenswrapper[2583]: W0713 22:13:23.489006 2583 feature_gate.go:328] unrecognized feature gate: Example2 Jul 13 22:13:25.251276 ip-10-0-133-179 kubenswrapper[2583]: W0713 22:13:23.489009 2583 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 13 22:13:25.251276 ip-10-0-133-179 kubenswrapper[2583]: W0713 22:13:23.489011 2583 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 13 22:13:25.251276 ip-10-0-133-179 kubenswrapper[2583]: W0713 22:13:23.489014 2583 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 13 22:13:25.251276 ip-10-0-133-179 kubenswrapper[2583]: W0713 22:13:23.489016 2583 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 13 22:13:25.251276 ip-10-0-133-179 kubenswrapper[2583]: W0713 22:13:23.489019 2583 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 13 22:13:25.251276 ip-10-0-133-179 kubenswrapper[2583]: W0713 22:13:23.489022 2583 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 13 22:13:25.251276 ip-10-0-133-179 kubenswrapper[2583]: W0713 22:13:23.489024 2583 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 13 22:13:25.251276 ip-10-0-133-179 kubenswrapper[2583]: W0713 22:13:23.489027 2583 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 13 22:13:25.251276 ip-10-0-133-179 kubenswrapper[2583]: W0713 22:13:23.489030 2583 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 13 22:13:25.251276 ip-10-0-133-179 kubenswrapper[2583]: W0713 22:13:23.489032 2583 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 13 22:13:25.251276 ip-10-0-133-179 kubenswrapper[2583]: W0713 22:13:23.489035 2583 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 13 22:13:25.252318 ip-10-0-133-179 kubenswrapper[2583]: W0713 22:13:23.489038 2583 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 13 22:13:25.252318 ip-10-0-133-179 kubenswrapper[2583]: W0713 22:13:23.489041 2583 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 13 22:13:25.252318 ip-10-0-133-179 kubenswrapper[2583]: W0713 22:13:23.489043 2583 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 13 22:13:25.252318 ip-10-0-133-179 kubenswrapper[2583]: W0713 22:13:23.489046 2583 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 13 22:13:25.252318 ip-10-0-133-179 kubenswrapper[2583]: W0713 22:13:23.489050 2583 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 13 22:13:25.252318 ip-10-0-133-179 kubenswrapper[2583]: W0713 22:13:23.489052 2583 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 13 22:13:25.252318 ip-10-0-133-179 kubenswrapper[2583]: W0713 22:13:23.489055 2583 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 13 22:13:25.252318 ip-10-0-133-179 kubenswrapper[2583]: W0713 22:13:23.489057 2583 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 13 22:13:25.252318 ip-10-0-133-179 kubenswrapper[2583]: W0713 22:13:23.489060 2583 feature_gate.go:328] unrecognized feature gate: Example Jul 13 22:13:25.252318 ip-10-0-133-179 kubenswrapper[2583]: W0713 22:13:23.489062 2583 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 13 22:13:25.252318 ip-10-0-133-179 kubenswrapper[2583]: W0713 22:13:23.489065 2583 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 13 22:13:25.252318 ip-10-0-133-179 kubenswrapper[2583]: W0713 22:13:23.489067 2583 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 13 22:13:25.252318 ip-10-0-133-179 kubenswrapper[2583]: W0713 22:13:23.489070 2583 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 13 22:13:25.252318 ip-10-0-133-179 kubenswrapper[2583]: W0713 22:13:23.489072 2583 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 13 22:13:25.252318 ip-10-0-133-179 kubenswrapper[2583]: W0713 22:13:23.489075 2583 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 13 22:13:25.252318 ip-10-0-133-179 kubenswrapper[2583]: W0713 22:13:23.489077 2583 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 13 22:13:25.252318 ip-10-0-133-179 kubenswrapper[2583]: W0713 22:13:23.489079 2583 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 13 22:13:25.252318 ip-10-0-133-179 kubenswrapper[2583]: W0713 22:13:23.489082 2583 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 13 22:13:25.252318 ip-10-0-133-179 kubenswrapper[2583]: W0713 22:13:23.489085 2583 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 13 22:13:25.252318 ip-10-0-133-179 kubenswrapper[2583]: W0713 22:13:23.489087 2583 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 13 22:13:25.253137 ip-10-0-133-179 kubenswrapper[2583]: W0713 22:13:23.489089 2583 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 13 22:13:25.253137 ip-10-0-133-179 kubenswrapper[2583]: W0713 22:13:23.489092 2583 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 13 22:13:25.253137 ip-10-0-133-179 kubenswrapper[2583]: W0713 22:13:23.489095 2583 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 13 22:13:25.253137 ip-10-0-133-179 kubenswrapper[2583]: W0713 22:13:23.489097 2583 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 13 22:13:25.253137 ip-10-0-133-179 kubenswrapper[2583]: W0713 22:13:23.489099 2583 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 13 22:13:25.253137 ip-10-0-133-179 kubenswrapper[2583]: W0713 22:13:23.489102 2583 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 13 22:13:25.253137 ip-10-0-133-179 kubenswrapper[2583]: W0713 22:13:23.489104 2583 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 13 22:13:25.253137 ip-10-0-133-179 kubenswrapper[2583]: W0713 22:13:23.489107 2583 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 13 22:13:25.253137 ip-10-0-133-179 kubenswrapper[2583]: W0713 22:13:23.489109 2583 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 13 22:13:25.253137 ip-10-0-133-179 kubenswrapper[2583]: W0713 22:13:23.489112 2583 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 13 22:13:25.253137 ip-10-0-133-179 kubenswrapper[2583]: W0713 22:13:23.489114 2583 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 13 22:13:25.253137 ip-10-0-133-179 kubenswrapper[2583]: W0713 22:13:23.489117 2583 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 13 22:13:25.253137 ip-10-0-133-179 kubenswrapper[2583]: W0713 22:13:23.489122 2583 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 13 22:13:25.253137 ip-10-0-133-179 kubenswrapper[2583]: W0713 22:13:23.489124 2583 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 13 22:13:25.253137 ip-10-0-133-179 kubenswrapper[2583]: W0713 22:13:23.489127 2583 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 13 22:13:25.253137 ip-10-0-133-179 kubenswrapper[2583]: W0713 22:13:23.489129 2583 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 13 22:13:25.253137 ip-10-0-133-179 kubenswrapper[2583]: W0713 22:13:23.489133 2583 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 13 22:13:25.253137 ip-10-0-133-179 kubenswrapper[2583]: W0713 22:13:23.489136 2583 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 13 22:13:25.253137 ip-10-0-133-179 kubenswrapper[2583]: W0713 22:13:23.489139 2583 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 13 22:13:25.253137 ip-10-0-133-179 kubenswrapper[2583]: W0713 22:13:23.489141 2583 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 13 22:13:25.253943 ip-10-0-133-179 kubenswrapper[2583]: W0713 22:13:23.489144 2583 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 13 22:13:25.253943 ip-10-0-133-179 kubenswrapper[2583]: W0713 22:13:23.489148 2583 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 13 22:13:25.253943 ip-10-0-133-179 kubenswrapper[2583]: W0713 22:13:23.489152 2583 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 13 22:13:25.253943 ip-10-0-133-179 kubenswrapper[2583]: W0713 22:13:23.489155 2583 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 13 22:13:25.253943 ip-10-0-133-179 kubenswrapper[2583]: W0713 22:13:23.489158 2583 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 13 22:13:25.253943 ip-10-0-133-179 kubenswrapper[2583]: W0713 22:13:23.489160 2583 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 13 22:13:25.253943 ip-10-0-133-179 kubenswrapper[2583]: W0713 22:13:23.489163 2583 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 13 22:13:25.253943 ip-10-0-133-179 kubenswrapper[2583]: W0713 22:13:23.489165 2583 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 13 22:13:25.253943 ip-10-0-133-179 kubenswrapper[2583]: W0713 22:13:23.489168 2583 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 13 22:13:25.253943 ip-10-0-133-179 kubenswrapper[2583]: W0713 22:13:23.489170 2583 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 13 22:13:25.253943 ip-10-0-133-179 kubenswrapper[2583]: W0713 22:13:23.489173 2583 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 13 22:13:25.253943 ip-10-0-133-179 kubenswrapper[2583]: W0713 22:13:23.489176 2583 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 13 22:13:25.253943 ip-10-0-133-179 kubenswrapper[2583]: W0713 22:13:23.489178 2583 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 13 22:13:25.253943 ip-10-0-133-179 kubenswrapper[2583]: W0713 22:13:23.489181 2583 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 13 22:13:25.253943 ip-10-0-133-179 kubenswrapper[2583]: W0713 22:13:23.489183 2583 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 13 22:13:25.253943 ip-10-0-133-179 kubenswrapper[2583]: W0713 22:13:23.489188 2583 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 13 22:13:25.253943 ip-10-0-133-179 kubenswrapper[2583]: W0713 22:13:23.489191 2583 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 13 22:13:25.253943 ip-10-0-133-179 kubenswrapper[2583]: W0713 22:13:23.489194 2583 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 13 22:13:25.253943 ip-10-0-133-179 kubenswrapper[2583]: W0713 22:13:23.489197 2583 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 13 22:13:25.254645 ip-10-0-133-179 kubenswrapper[2583]: W0713 22:13:23.489200 2583 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 13 22:13:25.254645 ip-10-0-133-179 kubenswrapper[2583]: W0713 22:13:23.489202 2583 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 13 22:13:25.254645 ip-10-0-133-179 kubenswrapper[2583]: W0713 22:13:23.489205 2583 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 13 22:13:25.254645 ip-10-0-133-179 kubenswrapper[2583]: W0713 22:13:23.489208 2583 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 13 22:13:25.254645 ip-10-0-133-179 kubenswrapper[2583]: W0713 22:13:23.489210 2583 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 13 22:13:25.254645 ip-10-0-133-179 kubenswrapper[2583]: W0713 22:13:23.489215 2583 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 13 22:13:25.254645 ip-10-0-133-179 kubenswrapper[2583]: W0713 22:13:23.489219 2583 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 13 22:13:25.254645 ip-10-0-133-179 kubenswrapper[2583]: W0713 22:13:23.489235 2583 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 13 22:13:25.254645 ip-10-0-133-179 kubenswrapper[2583]: W0713 22:13:23.489238 2583 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 13 22:13:25.254645 ip-10-0-133-179 kubenswrapper[2583]: W0713 22:13:23.489242 2583 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 13 22:13:25.254645 ip-10-0-133-179 kubenswrapper[2583]: W0713 22:13:23.489245 2583 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 13 22:13:25.254645 ip-10-0-133-179 kubenswrapper[2583]: W0713 22:13:23.489247 2583 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 13 22:13:25.254645 ip-10-0-133-179 kubenswrapper[2583]: W0713 22:13:23.489250 2583 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 13 22:13:25.254645 ip-10-0-133-179 kubenswrapper[2583]: W0713 22:13:23.489253 2583 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 13 22:13:25.254645 ip-10-0-133-179 kubenswrapper[2583]: W0713 22:13:23.489255 2583 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 13 22:13:25.254645 ip-10-0-133-179 kubenswrapper[2583]: W0713 22:13:23.489258 2583 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 13 22:13:25.254645 ip-10-0-133-179 kubenswrapper[2583]: W0713 22:13:23.489260 2583 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 13 22:13:25.254645 ip-10-0-133-179 kubenswrapper[2583]: W0713 22:13:23.489263 2583 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 13 22:13:25.255211 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:13:23.489268 2583 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jul 13 22:13:25.255211 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:13:23.495926 2583 server.go:529] "Kubelet version" kubeletVersion="v1.34.9" Jul 13 22:13:25.255211 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:13:23.495948 2583 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK="" Jul 13 22:13:25.255211 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:13:23.496820 2583 watchdog_linux.go:95] "Systemd watchdog is not enabled" Jul 13 22:13:25.255211 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:13:23.496827 2583 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started." Jul 13 22:13:25.255211 ip-10-0-133-179 kubenswrapper[2583]: W0713 22:13:23.496905 2583 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 13 22:13:25.255211 ip-10-0-133-179 kubenswrapper[2583]: W0713 22:13:23.496911 2583 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 13 22:13:25.255211 ip-10-0-133-179 kubenswrapper[2583]: W0713 22:13:23.496915 2583 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 13 22:13:25.255211 ip-10-0-133-179 kubenswrapper[2583]: W0713 22:13:23.496919 2583 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 13 22:13:25.255211 ip-10-0-133-179 kubenswrapper[2583]: W0713 22:13:23.496922 2583 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 13 22:13:25.255211 ip-10-0-133-179 kubenswrapper[2583]: W0713 22:13:23.496925 2583 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 13 22:13:25.255211 ip-10-0-133-179 kubenswrapper[2583]: W0713 22:13:23.496928 2583 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 13 22:13:25.255211 ip-10-0-133-179 kubenswrapper[2583]: W0713 22:13:23.496931 2583 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 13 22:13:25.255211 ip-10-0-133-179 kubenswrapper[2583]: W0713 22:13:23.496933 2583 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 13 22:13:25.255701 ip-10-0-133-179 kubenswrapper[2583]: W0713 22:13:23.496936 2583 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 13 22:13:25.255701 ip-10-0-133-179 kubenswrapper[2583]: W0713 22:13:23.496939 2583 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 13 22:13:25.255701 ip-10-0-133-179 kubenswrapper[2583]: W0713 22:13:23.496941 2583 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 13 22:13:25.255701 ip-10-0-133-179 kubenswrapper[2583]: W0713 22:13:23.496944 2583 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 13 22:13:25.255701 ip-10-0-133-179 kubenswrapper[2583]: W0713 22:13:23.496947 2583 feature_gate.go:328] unrecognized feature gate: Example Jul 13 22:13:25.255701 ip-10-0-133-179 kubenswrapper[2583]: W0713 22:13:23.496950 2583 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 13 22:13:25.255701 ip-10-0-133-179 kubenswrapper[2583]: W0713 22:13:23.496953 2583 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 13 22:13:25.255701 ip-10-0-133-179 kubenswrapper[2583]: W0713 22:13:23.496955 2583 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 13 22:13:25.255701 ip-10-0-133-179 kubenswrapper[2583]: W0713 22:13:23.496958 2583 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 13 22:13:25.255701 ip-10-0-133-179 kubenswrapper[2583]: W0713 22:13:23.496961 2583 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 13 22:13:25.255701 ip-10-0-133-179 kubenswrapper[2583]: W0713 22:13:23.496963 2583 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 13 22:13:25.255701 ip-10-0-133-179 kubenswrapper[2583]: W0713 22:13:23.496966 2583 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 13 22:13:25.255701 ip-10-0-133-179 kubenswrapper[2583]: W0713 22:13:23.496969 2583 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 13 22:13:25.255701 ip-10-0-133-179 kubenswrapper[2583]: W0713 22:13:23.496971 2583 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 13 22:13:25.255701 ip-10-0-133-179 kubenswrapper[2583]: W0713 22:13:23.496974 2583 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 13 22:13:25.255701 ip-10-0-133-179 kubenswrapper[2583]: W0713 22:13:23.496977 2583 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 13 22:13:25.255701 ip-10-0-133-179 kubenswrapper[2583]: W0713 22:13:23.496980 2583 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 13 22:13:25.255701 ip-10-0-133-179 kubenswrapper[2583]: W0713 22:13:23.496982 2583 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 13 22:13:25.255701 ip-10-0-133-179 kubenswrapper[2583]: W0713 22:13:23.496985 2583 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 13 22:13:25.255701 ip-10-0-133-179 kubenswrapper[2583]: W0713 22:13:23.496988 2583 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 13 22:13:25.256337 ip-10-0-133-179 kubenswrapper[2583]: W0713 22:13:23.496992 2583 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 13 22:13:25.256337 ip-10-0-133-179 kubenswrapper[2583]: W0713 22:13:23.496994 2583 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 13 22:13:25.256337 ip-10-0-133-179 kubenswrapper[2583]: W0713 22:13:23.496997 2583 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 13 22:13:25.256337 ip-10-0-133-179 kubenswrapper[2583]: W0713 22:13:23.497000 2583 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 13 22:13:25.256337 ip-10-0-133-179 kubenswrapper[2583]: W0713 22:13:23.497003 2583 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 13 22:13:25.256337 ip-10-0-133-179 kubenswrapper[2583]: W0713 22:13:23.497006 2583 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 13 22:13:25.256337 ip-10-0-133-179 kubenswrapper[2583]: W0713 22:13:23.497008 2583 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 13 22:13:25.256337 ip-10-0-133-179 kubenswrapper[2583]: W0713 22:13:23.497010 2583 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 13 22:13:25.256337 ip-10-0-133-179 kubenswrapper[2583]: W0713 22:13:23.497013 2583 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 13 22:13:25.256337 ip-10-0-133-179 kubenswrapper[2583]: W0713 22:13:23.497015 2583 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 13 22:13:25.256337 ip-10-0-133-179 kubenswrapper[2583]: W0713 22:13:23.497018 2583 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 13 22:13:25.256337 ip-10-0-133-179 kubenswrapper[2583]: W0713 22:13:23.497020 2583 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 13 22:13:25.256337 ip-10-0-133-179 kubenswrapper[2583]: W0713 22:13:23.497023 2583 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 13 22:13:25.256337 ip-10-0-133-179 kubenswrapper[2583]: W0713 22:13:23.497026 2583 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 13 22:13:25.256337 ip-10-0-133-179 kubenswrapper[2583]: W0713 22:13:23.497029 2583 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 13 22:13:25.256337 ip-10-0-133-179 kubenswrapper[2583]: W0713 22:13:23.497031 2583 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 13 22:13:25.256337 ip-10-0-133-179 kubenswrapper[2583]: W0713 22:13:23.497034 2583 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 13 22:13:25.256337 ip-10-0-133-179 kubenswrapper[2583]: W0713 22:13:23.497037 2583 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 13 22:13:25.256337 ip-10-0-133-179 kubenswrapper[2583]: W0713 22:13:23.497041 2583 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 13 22:13:25.256963 ip-10-0-133-179 kubenswrapper[2583]: W0713 22:13:23.497045 2583 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 13 22:13:25.256963 ip-10-0-133-179 kubenswrapper[2583]: W0713 22:13:23.497048 2583 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 13 22:13:25.256963 ip-10-0-133-179 kubenswrapper[2583]: W0713 22:13:23.497051 2583 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 13 22:13:25.256963 ip-10-0-133-179 kubenswrapper[2583]: W0713 22:13:23.497053 2583 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 13 22:13:25.256963 ip-10-0-133-179 kubenswrapper[2583]: W0713 22:13:23.497056 2583 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 13 22:13:25.256963 ip-10-0-133-179 kubenswrapper[2583]: W0713 22:13:23.497058 2583 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 13 22:13:25.256963 ip-10-0-133-179 kubenswrapper[2583]: W0713 22:13:23.497061 2583 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 13 22:13:25.256963 ip-10-0-133-179 kubenswrapper[2583]: W0713 22:13:23.497063 2583 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 13 22:13:25.256963 ip-10-0-133-179 kubenswrapper[2583]: W0713 22:13:23.497066 2583 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 13 22:13:25.256963 ip-10-0-133-179 kubenswrapper[2583]: W0713 22:13:23.497068 2583 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 13 22:13:25.256963 ip-10-0-133-179 kubenswrapper[2583]: W0713 22:13:23.497071 2583 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 13 22:13:25.256963 ip-10-0-133-179 kubenswrapper[2583]: W0713 22:13:23.497073 2583 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 13 22:13:25.256963 ip-10-0-133-179 kubenswrapper[2583]: W0713 22:13:23.497076 2583 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 13 22:13:25.256963 ip-10-0-133-179 kubenswrapper[2583]: W0713 22:13:23.497080 2583 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 13 22:13:25.256963 ip-10-0-133-179 kubenswrapper[2583]: W0713 22:13:23.497084 2583 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 13 22:13:25.256963 ip-10-0-133-179 kubenswrapper[2583]: W0713 22:13:23.497087 2583 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 13 22:13:25.256963 ip-10-0-133-179 kubenswrapper[2583]: W0713 22:13:23.497090 2583 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 13 22:13:25.256963 ip-10-0-133-179 kubenswrapper[2583]: W0713 22:13:23.497093 2583 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 13 22:13:25.256963 ip-10-0-133-179 kubenswrapper[2583]: W0713 22:13:23.497095 2583 feature_gate.go:328] unrecognized feature gate: Example2 Jul 13 22:13:25.257588 ip-10-0-133-179 kubenswrapper[2583]: W0713 22:13:23.497098 2583 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 13 22:13:25.257588 ip-10-0-133-179 kubenswrapper[2583]: W0713 22:13:23.497101 2583 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 13 22:13:25.257588 ip-10-0-133-179 kubenswrapper[2583]: W0713 22:13:23.497104 2583 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 13 22:13:25.257588 ip-10-0-133-179 kubenswrapper[2583]: W0713 22:13:23.497106 2583 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 13 22:13:25.257588 ip-10-0-133-179 kubenswrapper[2583]: W0713 22:13:23.497109 2583 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 13 22:13:25.257588 ip-10-0-133-179 kubenswrapper[2583]: W0713 22:13:23.497111 2583 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 13 22:13:25.257588 ip-10-0-133-179 kubenswrapper[2583]: W0713 22:13:23.497114 2583 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 13 22:13:25.257588 ip-10-0-133-179 kubenswrapper[2583]: W0713 22:13:23.497117 2583 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 13 22:13:25.257588 ip-10-0-133-179 kubenswrapper[2583]: W0713 22:13:23.497120 2583 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 13 22:13:25.257588 ip-10-0-133-179 kubenswrapper[2583]: W0713 22:13:23.497122 2583 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 13 22:13:25.257588 ip-10-0-133-179 kubenswrapper[2583]: W0713 22:13:23.497125 2583 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 13 22:13:25.257588 ip-10-0-133-179 kubenswrapper[2583]: W0713 22:13:23.497128 2583 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 13 22:13:25.257588 ip-10-0-133-179 kubenswrapper[2583]: W0713 22:13:23.497131 2583 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 13 22:13:25.257588 ip-10-0-133-179 kubenswrapper[2583]: W0713 22:13:23.497133 2583 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 13 22:13:25.257588 ip-10-0-133-179 kubenswrapper[2583]: W0713 22:13:23.497136 2583 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 13 22:13:25.257588 ip-10-0-133-179 kubenswrapper[2583]: W0713 22:13:23.497138 2583 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 13 22:13:25.257588 ip-10-0-133-179 kubenswrapper[2583]: W0713 22:13:23.497141 2583 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 13 22:13:25.257588 ip-10-0-133-179 kubenswrapper[2583]: W0713 22:13:23.497143 2583 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 13 22:13:25.257588 ip-10-0-133-179 kubenswrapper[2583]: W0713 22:13:23.497146 2583 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 13 22:13:25.257588 ip-10-0-133-179 kubenswrapper[2583]: W0713 22:13:23.497149 2583 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 13 22:13:25.258222 ip-10-0-133-179 kubenswrapper[2583]: W0713 22:13:23.497151 2583 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 13 22:13:25.258222 ip-10-0-133-179 kubenswrapper[2583]: W0713 22:13:23.497154 2583 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 13 22:13:25.258222 ip-10-0-133-179 kubenswrapper[2583]: W0713 22:13:23.497159 2583 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 13 22:13:25.258222 ip-10-0-133-179 kubenswrapper[2583]: W0713 22:13:23.497162 2583 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 13 22:13:25.258222 ip-10-0-133-179 kubenswrapper[2583]: W0713 22:13:23.497165 2583 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 13 22:13:25.258222 ip-10-0-133-179 kubenswrapper[2583]: W0713 22:13:23.497170 2583 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 13 22:13:25.258222 ip-10-0-133-179 kubenswrapper[2583]: W0713 22:13:23.497173 2583 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 13 22:13:25.258222 ip-10-0-133-179 kubenswrapper[2583]: W0713 22:13:23.497176 2583 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 13 22:13:25.258222 ip-10-0-133-179 kubenswrapper[2583]: W0713 22:13:23.497179 2583 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 13 22:13:25.258222 ip-10-0-133-179 kubenswrapper[2583]: W0713 22:13:23.497182 2583 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 13 22:13:25.258222 ip-10-0-133-179 kubenswrapper[2583]: W0713 22:13:23.497184 2583 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 13 22:13:25.258222 ip-10-0-133-179 kubenswrapper[2583]: W0713 22:13:23.497187 2583 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 13 22:13:25.258222 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:13:23.497192 2583 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jul 13 22:13:25.258222 ip-10-0-133-179 kubenswrapper[2583]: W0713 22:13:23.497326 2583 feature_gate.go:328] unrecognized feature gate: Example Jul 13 22:13:25.258222 ip-10-0-133-179 kubenswrapper[2583]: W0713 22:13:23.497332 2583 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 13 22:13:25.258751 ip-10-0-133-179 kubenswrapper[2583]: W0713 22:13:23.497334 2583 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 13 22:13:25.258751 ip-10-0-133-179 kubenswrapper[2583]: W0713 22:13:23.497337 2583 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 13 22:13:25.258751 ip-10-0-133-179 kubenswrapper[2583]: W0713 22:13:23.497340 2583 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 13 22:13:25.258751 ip-10-0-133-179 kubenswrapper[2583]: W0713 22:13:23.497343 2583 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 13 22:13:25.258751 ip-10-0-133-179 kubenswrapper[2583]: W0713 22:13:23.497348 2583 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 13 22:13:25.258751 ip-10-0-133-179 kubenswrapper[2583]: W0713 22:13:23.497350 2583 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 13 22:13:25.258751 ip-10-0-133-179 kubenswrapper[2583]: W0713 22:13:23.497353 2583 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 13 22:13:25.258751 ip-10-0-133-179 kubenswrapper[2583]: W0713 22:13:23.497355 2583 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 13 22:13:25.258751 ip-10-0-133-179 kubenswrapper[2583]: W0713 22:13:23.497358 2583 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 13 22:13:25.258751 ip-10-0-133-179 kubenswrapper[2583]: W0713 22:13:23.497360 2583 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 13 22:13:25.258751 ip-10-0-133-179 kubenswrapper[2583]: W0713 22:13:23.497363 2583 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 13 22:13:25.258751 ip-10-0-133-179 kubenswrapper[2583]: W0713 22:13:23.497365 2583 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 13 22:13:25.258751 ip-10-0-133-179 kubenswrapper[2583]: W0713 22:13:23.497370 2583 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 13 22:13:25.258751 ip-10-0-133-179 kubenswrapper[2583]: W0713 22:13:23.497374 2583 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 13 22:13:25.258751 ip-10-0-133-179 kubenswrapper[2583]: W0713 22:13:23.497377 2583 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 13 22:13:25.258751 ip-10-0-133-179 kubenswrapper[2583]: W0713 22:13:23.497380 2583 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 13 22:13:25.258751 ip-10-0-133-179 kubenswrapper[2583]: W0713 22:13:23.497383 2583 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 13 22:13:25.258751 ip-10-0-133-179 kubenswrapper[2583]: W0713 22:13:23.497385 2583 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 13 22:13:25.258751 ip-10-0-133-179 kubenswrapper[2583]: W0713 22:13:23.497388 2583 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 13 22:13:25.259331 ip-10-0-133-179 kubenswrapper[2583]: W0713 22:13:23.497391 2583 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 13 22:13:25.259331 ip-10-0-133-179 kubenswrapper[2583]: W0713 22:13:23.497395 2583 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 13 22:13:25.259331 ip-10-0-133-179 kubenswrapper[2583]: W0713 22:13:23.497397 2583 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 13 22:13:25.259331 ip-10-0-133-179 kubenswrapper[2583]: W0713 22:13:23.497400 2583 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 13 22:13:25.259331 ip-10-0-133-179 kubenswrapper[2583]: W0713 22:13:23.497403 2583 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 13 22:13:25.259331 ip-10-0-133-179 kubenswrapper[2583]: W0713 22:13:23.497406 2583 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 13 22:13:25.259331 ip-10-0-133-179 kubenswrapper[2583]: W0713 22:13:23.497409 2583 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 13 22:13:25.259331 ip-10-0-133-179 kubenswrapper[2583]: W0713 22:13:23.497412 2583 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 13 22:13:25.259331 ip-10-0-133-179 kubenswrapper[2583]: W0713 22:13:23.497414 2583 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 13 22:13:25.259331 ip-10-0-133-179 kubenswrapper[2583]: W0713 22:13:23.497417 2583 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 13 22:13:25.259331 ip-10-0-133-179 kubenswrapper[2583]: W0713 22:13:23.497419 2583 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 13 22:13:25.259331 ip-10-0-133-179 kubenswrapper[2583]: W0713 22:13:23.497423 2583 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 13 22:13:25.259331 ip-10-0-133-179 kubenswrapper[2583]: W0713 22:13:23.497427 2583 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 13 22:13:25.259331 ip-10-0-133-179 kubenswrapper[2583]: W0713 22:13:23.497430 2583 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 13 22:13:25.259331 ip-10-0-133-179 kubenswrapper[2583]: W0713 22:13:23.497432 2583 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 13 22:13:25.259331 ip-10-0-133-179 kubenswrapper[2583]: W0713 22:13:23.497435 2583 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 13 22:13:25.259331 ip-10-0-133-179 kubenswrapper[2583]: W0713 22:13:23.497437 2583 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 13 22:13:25.259331 ip-10-0-133-179 kubenswrapper[2583]: W0713 22:13:23.497441 2583 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 13 22:13:25.259331 ip-10-0-133-179 kubenswrapper[2583]: W0713 22:13:23.497443 2583 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 13 22:13:25.260076 ip-10-0-133-179 kubenswrapper[2583]: W0713 22:13:23.497446 2583 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 13 22:13:25.260076 ip-10-0-133-179 kubenswrapper[2583]: W0713 22:13:23.497448 2583 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 13 22:13:25.260076 ip-10-0-133-179 kubenswrapper[2583]: W0713 22:13:23.497451 2583 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 13 22:13:25.260076 ip-10-0-133-179 kubenswrapper[2583]: W0713 22:13:23.497453 2583 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 13 22:13:25.260076 ip-10-0-133-179 kubenswrapper[2583]: W0713 22:13:23.497456 2583 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 13 22:13:25.260076 ip-10-0-133-179 kubenswrapper[2583]: W0713 22:13:23.497458 2583 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 13 22:13:25.260076 ip-10-0-133-179 kubenswrapper[2583]: W0713 22:13:23.497461 2583 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 13 22:13:25.260076 ip-10-0-133-179 kubenswrapper[2583]: W0713 22:13:23.497464 2583 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 13 22:13:25.260076 ip-10-0-133-179 kubenswrapper[2583]: W0713 22:13:23.497466 2583 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 13 22:13:25.260076 ip-10-0-133-179 kubenswrapper[2583]: W0713 22:13:23.497468 2583 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 13 22:13:25.260076 ip-10-0-133-179 kubenswrapper[2583]: W0713 22:13:23.497471 2583 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 13 22:13:25.260076 ip-10-0-133-179 kubenswrapper[2583]: W0713 22:13:23.497474 2583 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 13 22:13:25.260076 ip-10-0-133-179 kubenswrapper[2583]: W0713 22:13:23.497476 2583 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 13 22:13:25.260076 ip-10-0-133-179 kubenswrapper[2583]: W0713 22:13:23.497479 2583 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 13 22:13:25.260076 ip-10-0-133-179 kubenswrapper[2583]: W0713 22:13:23.497483 2583 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 13 22:13:25.260076 ip-10-0-133-179 kubenswrapper[2583]: W0713 22:13:23.497486 2583 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 13 22:13:25.260076 ip-10-0-133-179 kubenswrapper[2583]: W0713 22:13:23.497488 2583 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 13 22:13:25.260076 ip-10-0-133-179 kubenswrapper[2583]: W0713 22:13:23.497491 2583 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 13 22:13:25.260076 ip-10-0-133-179 kubenswrapper[2583]: W0713 22:13:23.497493 2583 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 13 22:13:25.260076 ip-10-0-133-179 kubenswrapper[2583]: W0713 22:13:23.497495 2583 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 13 22:13:25.261322 ip-10-0-133-179 kubenswrapper[2583]: W0713 22:13:23.497498 2583 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 13 22:13:25.261322 ip-10-0-133-179 kubenswrapper[2583]: W0713 22:13:23.497501 2583 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 13 22:13:25.261322 ip-10-0-133-179 kubenswrapper[2583]: W0713 22:13:23.497503 2583 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 13 22:13:25.261322 ip-10-0-133-179 kubenswrapper[2583]: W0713 22:13:23.497507 2583 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 13 22:13:25.261322 ip-10-0-133-179 kubenswrapper[2583]: W0713 22:13:23.497511 2583 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 13 22:13:25.261322 ip-10-0-133-179 kubenswrapper[2583]: W0713 22:13:23.497514 2583 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 13 22:13:25.261322 ip-10-0-133-179 kubenswrapper[2583]: W0713 22:13:23.497517 2583 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 13 22:13:25.261322 ip-10-0-133-179 kubenswrapper[2583]: W0713 22:13:23.497520 2583 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 13 22:13:25.261322 ip-10-0-133-179 kubenswrapper[2583]: W0713 22:13:23.497523 2583 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 13 22:13:25.261322 ip-10-0-133-179 kubenswrapper[2583]: W0713 22:13:23.497525 2583 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 13 22:13:25.261322 ip-10-0-133-179 kubenswrapper[2583]: W0713 22:13:23.497529 2583 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 13 22:13:25.261322 ip-10-0-133-179 kubenswrapper[2583]: W0713 22:13:23.497531 2583 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 13 22:13:25.261322 ip-10-0-133-179 kubenswrapper[2583]: W0713 22:13:23.497534 2583 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 13 22:13:25.261322 ip-10-0-133-179 kubenswrapper[2583]: W0713 22:13:23.497537 2583 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 13 22:13:25.261322 ip-10-0-133-179 kubenswrapper[2583]: W0713 22:13:23.497539 2583 feature_gate.go:328] unrecognized feature gate: Example2 Jul 13 22:13:25.261322 ip-10-0-133-179 kubenswrapper[2583]: W0713 22:13:23.497541 2583 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 13 22:13:25.261322 ip-10-0-133-179 kubenswrapper[2583]: W0713 22:13:23.497544 2583 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 13 22:13:25.261322 ip-10-0-133-179 kubenswrapper[2583]: W0713 22:13:23.497547 2583 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 13 22:13:25.261322 ip-10-0-133-179 kubenswrapper[2583]: W0713 22:13:23.497549 2583 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 13 22:13:25.261322 ip-10-0-133-179 kubenswrapper[2583]: W0713 22:13:23.497552 2583 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 13 22:13:25.261946 ip-10-0-133-179 kubenswrapper[2583]: W0713 22:13:23.497554 2583 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 13 22:13:25.261946 ip-10-0-133-179 kubenswrapper[2583]: W0713 22:13:23.497557 2583 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 13 22:13:25.261946 ip-10-0-133-179 kubenswrapper[2583]: W0713 22:13:23.497559 2583 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 13 22:13:25.261946 ip-10-0-133-179 kubenswrapper[2583]: W0713 22:13:23.497562 2583 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 13 22:13:25.261946 ip-10-0-133-179 kubenswrapper[2583]: W0713 22:13:23.497564 2583 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 13 22:13:25.261946 ip-10-0-133-179 kubenswrapper[2583]: W0713 22:13:23.497567 2583 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 13 22:13:25.261946 ip-10-0-133-179 kubenswrapper[2583]: W0713 22:13:23.497571 2583 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 13 22:13:25.261946 ip-10-0-133-179 kubenswrapper[2583]: W0713 22:13:23.497573 2583 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 13 22:13:25.261946 ip-10-0-133-179 kubenswrapper[2583]: W0713 22:13:23.497576 2583 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 13 22:13:25.261946 ip-10-0-133-179 kubenswrapper[2583]: W0713 22:13:23.497578 2583 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 13 22:13:25.261946 ip-10-0-133-179 kubenswrapper[2583]: W0713 22:13:23.497581 2583 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 13 22:13:25.261946 ip-10-0-133-179 kubenswrapper[2583]: W0713 22:13:23.497583 2583 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 13 22:13:25.261946 ip-10-0-133-179 kubenswrapper[2583]: W0713 22:13:23.497586 2583 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 13 22:13:25.261946 ip-10-0-133-179 kubenswrapper[2583]: W0713 22:13:23.497589 2583 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 13 22:13:25.261946 ip-10-0-133-179 kubenswrapper[2583]: W0713 22:13:23.497592 2583 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 13 22:13:25.261946 ip-10-0-133-179 kubenswrapper[2583]: W0713 22:13:23.497594 2583 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 13 22:13:25.261946 ip-10-0-133-179 kubenswrapper[2583]: W0713 22:13:23.497596 2583 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 13 22:13:25.261946 ip-10-0-133-179 kubenswrapper[2583]: W0713 22:13:23.497599 2583 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 13 22:13:25.261946 ip-10-0-133-179 kubenswrapper[2583]: W0713 22:13:23.497601 2583 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 13 22:13:25.263164 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:13:23.497606 2583 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jul 13 22:13:25.263164 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:13:23.497722 2583 server.go:962] "Client rotation is on, will bootstrap in background" Jul 13 22:13:25.263164 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:13:23.504606 2583 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir" Jul 13 22:13:25.263164 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:13:23.506042 2583 server.go:1019] "Starting client certificate rotation" Jul 13 22:13:25.263164 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:13:23.506537 2583 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet" Jul 13 22:13:25.263164 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:13:23.507196 2583 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet" Jul 13 22:13:25.263164 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:13:23.508954 2583 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Jul 13 22:13:25.263164 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:13:25.113526 2583 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Jul 13 22:13:25.263164 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:13:25.124535 2583 log.go:25] "Validated CRI v1 runtime API" Jul 13 22:13:25.263164 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:13:25.126878 2583 log.go:25] "Validated CRI v1 image API" Jul 13 22:13:25.263164 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:13:25.127621 2583 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd" Jul 13 22:13:25.263164 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:13:25.132620 2583 fs.go:135] Filesystem UUIDs: map[4da4cb24-5360-481e-be10-f95e8b4a9915:/dev/nvme0n1p3 7B77-95E7:/dev/nvme0n1p2 9370e4e1-dec5-41d0-88db-24e8d9e73b0c:/dev/nvme0n1p4] Jul 13 22:13:25.263541 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:13:25.132636 2583 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}] Jul 13 22:13:25.263541 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:13:25.134381 2583 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Jul 13 22:13:25.263856 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:13:25.136754 2583 manager.go:216] Machine: {Timestamp:2026-07-13 22:13:25.136276792 +0000 UTC m=+0.384192133 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3102472 MemoryCapacity:33164500992 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec2566a686f668fd9283a9578b3d1d99 SystemUUID:ec2566a6-86f6-68fd-9283-a9578b3d1d99 BootID:0d8ac528-56e3-41db-bf3e-e078c50b36b9 Filesystems:[{Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16582250496 Type:vfs Inodes:4048401 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6632902656 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6107136 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16582250496 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:fc:13:bf:56:19 Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:fc:13:bf:56:19 Speed:0 Mtu:9001} {Name:ovs-system MacAddress:ba:24:dc:85:df:50 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164500992 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None} Jul 13 22:13:25.263856 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:13:25.136847 2583 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available. Jul 13 22:13:25.263856 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:13:25.136978 2583 manager.go:232] Version: {KernelVersion:5.14.0-570.126.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260702-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:} Jul 13 22:13:25.263856 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:13:25.140954 2583 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[] Jul 13 22:13:25.263856 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:13:25.140983 2583 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-133-179.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2} Jul 13 22:13:25.263856 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:13:25.141771 2583 topology_manager.go:138] "Creating topology manager with none policy" Jul 13 22:13:25.263856 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:13:25.141781 2583 container_manager_linux.go:309] "Creating device plugin manager" Jul 13 22:13:25.263856 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:13:25.141790 2583 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock" Jul 13 22:13:25.263856 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:13:25.142390 2583 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock" Jul 13 22:13:25.263856 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:13:25.143724 2583 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager" Jul 13 22:13:25.263856 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:13:25.146885 2583 state_mem.go:36] "Initialized new in-memory state store" Jul 13 22:13:25.263856 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:13:25.146998 2583 server.go:1240] "Using root directory" path="/var/lib/kubelet" Jul 13 22:13:25.263856 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:13:25.147855 2583 kubelet.go:486] "Attempting to sync node with API server" Jul 13 22:13:25.263856 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:13:25.147872 2583 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests" Jul 13 22:13:25.263856 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:13:25.147893 2583 file.go:69] "Watching path" path="/etc/kubernetes/manifests" Jul 13 22:13:25.263856 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:13:25.147905 2583 kubelet.go:398] "Adding apiserver pod source" Jul 13 22:13:25.263856 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:13:25.147917 2583 apiserver.go:42] "Waiting for node sync before watching apiserver pods" Jul 13 22:13:25.263856 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:13:25.148365 2583 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Jul 13 22:13:25.264361 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:13:25.148385 2583 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Jul 13 22:13:25.264361 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:13:25.155702 2583 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.10-2.rhaos4.21.git83ee25b.el9" apiVersion="v1" Jul 13 22:13:25.264361 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:13:25.157668 2583 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled" Jul 13 22:13:25.264361 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:13:25.157696 2583 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled" Jul 13 22:13:25.264361 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:13:25.157884 2583 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume" Jul 13 22:13:25.264361 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:13:25.157909 2583 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir" Jul 13 22:13:25.264361 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:13:25.157918 2583 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo" Jul 13 22:13:25.264361 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:13:25.157926 2583 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path" Jul 13 22:13:25.264361 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:13:25.157935 2583 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs" Jul 13 22:13:25.264361 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:13:25.157940 2583 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret" Jul 13 22:13:25.264361 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:13:25.158494 2583 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi" Jul 13 22:13:25.264361 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:13:25.158501 2583 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api" Jul 13 22:13:25.264361 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:13:25.158508 2583 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc" Jul 13 22:13:25.264361 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:13:25.158514 2583 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap" Jul 13 22:13:25.264361 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:13:25.158527 2583 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected" Jul 13 22:13:25.264361 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:13:25.158536 2583 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume" Jul 13 22:13:25.264361 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:13:25.160276 2583 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi" Jul 13 22:13:25.264361 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:13:25.160283 2583 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image" Jul 13 22:13:25.264361 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:13:25.162768 2583 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-ddwqc" Jul 13 22:13:25.264361 ip-10-0-133-179 kubenswrapper[2583]: E0713 22:13:25.168792 2583 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service" Jul 13 22:13:25.264361 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:13:25.168825 2583 server.go:1268] "Started kubelet" Jul 13 22:13:25.264361 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:13:25.168903 2583 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-133-179.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Jul 13 22:13:25.264361 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:13:25.168959 2583 server.go:180] "Starting to listen" address="0.0.0.0" port=10250 Jul 13 22:13:25.264361 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:13:25.168980 2583 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10 Jul 13 22:13:25.264361 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:13:25.169031 2583 server_v1.go:49] "podresources" method="list" useActivePods=true Jul 13 22:13:25.264361 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:13:25.170599 2583 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock" Jul 13 22:13:25.264361 ip-10-0-133-179 kubenswrapper[2583]: E0713 22:13:25.171424 2583 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-133-179.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node" Jul 13 22:13:25.264361 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:13:25.171489 2583 server.go:310] "Adding debug handlers to kubelet server" Jul 13 22:13:25.264361 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:13:25.171817 2583 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-ddwqc" Jul 13 22:13:25.264361 ip-10-0-133-179 kubenswrapper[2583]: E0713 22:13:25.173415 2583 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache" Jul 13 22:13:25.264361 ip-10-0-133-179 kubenswrapper[2583]: E0713 22:13:25.173773 2583 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-133-179.ec2.internal.18c1f91722337f8d default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-133-179.ec2.internal,UID:ip-10-0-133-179.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-133-179.ec2.internal,},FirstTimestamp:2026-07-13 22:13:25.168791437 +0000 UTC m=+0.416706780,LastTimestamp:2026-07-13 22:13:25.168791437 +0000 UTC m=+0.416706780,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-133-179.ec2.internal,}" Jul 13 22:13:25.264361 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:13:25.176345 2583 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer" Jul 13 22:13:25.264361 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:13:25.176418 2583 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving" Jul 13 22:13:25.264361 ip-10-0-133-179 kubenswrapper[2583]: E0713 22:13:25.176430 2583 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-133-179.ec2.internal\" not found" Jul 13 22:13:25.264361 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:13:25.176927 2583 volume_manager.go:311] "The desired_state_of_world populator starts" Jul 13 22:13:25.264361 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:13:25.176949 2583 volume_manager.go:313] "Starting Kubelet Volume Manager" Jul 13 22:13:25.264361 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:13:25.176928 2583 desired_state_of_world_populator.go:146] "Desired state populator starts to run" Jul 13 22:13:25.264361 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:13:25.177060 2583 reconstruct.go:97] "Volume reconstruction finished" Jul 13 22:13:25.264361 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:13:25.177068 2583 reconciler.go:29] "Reconciler: start to sync state" Jul 13 22:13:25.265432 ip-10-0-133-179 kubenswrapper[2583]: E0713 22:13:25.178433 2583 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-133-179.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms" Jul 13 22:13:25.265432 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:13:25.178500 2583 factory.go:153] Registering CRI-O factory Jul 13 22:13:25.265432 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:13:25.178513 2583 factory.go:223] Registration of the crio container factory successfully Jul 13 22:13:25.265432 ip-10-0-133-179 kubenswrapper[2583]: E0713 22:13:25.178511 2583 reflector.go:205] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver" Jul 13 22:13:25.265432 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:13:25.178552 2583 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory Jul 13 22:13:25.265432 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:13:25.178559 2583 factory.go:55] Registering systemd factory Jul 13 22:13:25.265432 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:13:25.178564 2583 factory.go:223] Registration of the systemd container factory successfully Jul 13 22:13:25.265432 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:13:25.178599 2583 factory.go:103] Registering Raw factory Jul 13 22:13:25.265432 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:13:25.178611 2583 manager.go:1196] Started watching for new ooms in manager Jul 13 22:13:25.265432 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:13:25.179077 2583 manager.go:349] Starting recovery of all containers Jul 13 22:13:25.265432 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:13:25.189539 2583 manager.go:354] Recovery completed Jul 13 22:13:25.265432 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:13:25.195791 2583 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 13 22:13:25.265432 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:13:25.204971 2583 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-179.ec2.internal" event="NodeHasSufficientMemory" Jul 13 22:13:25.265432 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:13:25.205001 2583 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-179.ec2.internal" event="NodeHasNoDiskPressure" Jul 13 22:13:25.265432 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:13:25.205011 2583 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-179.ec2.internal" event="NodeHasSufficientPID" Jul 13 22:13:25.265432 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:13:25.205513 2583 cpu_manager.go:222] "Starting CPU manager" policy="none" Jul 13 22:13:25.265432 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:13:25.205524 2583 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s" Jul 13 22:13:25.265432 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:13:25.205540 2583 state_mem.go:36] "Initialized new in-memory state store" Jul 13 22:13:25.265432 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:13:25.205548 2583 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 13 22:13:25.265432 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:13:25.206357 2583 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-179.ec2.internal" event="NodeHasSufficientMemory" Jul 13 22:13:25.265432 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:13:25.206391 2583 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-179.ec2.internal" event="NodeHasNoDiskPressure" Jul 13 22:13:25.265432 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:13:25.206406 2583 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-179.ec2.internal" event="NodeHasSufficientPID" Jul 13 22:13:25.265432 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:13:25.206712 2583 policy_none.go:49] "None policy: Start" Jul 13 22:13:25.265432 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:13:25.206724 2583 memory_manager.go:187] "Starting memorymanager" policy="None" Jul 13 22:13:25.265432 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:13:25.206734 2583 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint" Jul 13 22:13:25.265432 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:13:25.207908 2583 policy_none.go:47] "Start" Jul 13 22:13:25.265432 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:13:25.246214 2583 manager.go:337] "Starting Device Plugin manager" Jul 13 22:13:25.265432 ip-10-0-133-179 kubenswrapper[2583]: E0713 22:13:25.246410 2583 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint" Jul 13 22:13:25.265432 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:13:25.246422 2583 server.go:87] "Starting device plugin registration server" Jul 13 22:13:25.265432 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:13:25.246730 2583 eviction_manager.go:189] "Eviction manager: starting control loop" Jul 13 22:13:25.265432 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:13:25.246747 2583 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s" Jul 13 22:13:25.265432 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:13:25.246903 2583 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry" Jul 13 22:13:25.265432 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:13:25.246996 2583 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts" Jul 13 22:13:25.265432 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:13:25.247007 2583 plugin_manager.go:118] "Starting Kubelet Plugin Manager" Jul 13 22:13:25.265432 ip-10-0-133-179 kubenswrapper[2583]: E0713 22:13:25.247492 2583 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\"" Jul 13 22:13:25.265432 ip-10-0-133-179 kubenswrapper[2583]: E0713 22:13:25.247542 2583 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-133-179.ec2.internal\" not found" Jul 13 22:13:25.424747 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:13:25.347860 2583 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 13 22:13:25.424747 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:13:25.348871 2583 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-179.ec2.internal" event="NodeHasSufficientMemory" Jul 13 22:13:25.424747 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:13:25.348900 2583 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-179.ec2.internal" event="NodeHasNoDiskPressure" Jul 13 22:13:25.424747 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:13:25.348915 2583 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-179.ec2.internal" event="NodeHasSufficientPID" Jul 13 22:13:25.424747 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:13:25.348942 2583 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-133-179.ec2.internal" Jul 13 22:13:25.424747 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:13:25.356919 2583 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-133-179.ec2.internal" Jul 13 22:13:25.424747 ip-10-0-133-179 kubenswrapper[2583]: E0713 22:13:25.356942 2583 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-133-179.ec2.internal\": node \"ip-10-0-133-179.ec2.internal\" not found" Jul 13 22:13:25.424747 ip-10-0-133-179 kubenswrapper[2583]: E0713 22:13:25.373767 2583 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-133-179.ec2.internal\" not found" Jul 13 22:13:25.424747 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:13:25.382083 2583 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4" Jul 13 22:13:25.424747 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:13:25.383365 2583 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6" Jul 13 22:13:25.424747 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:13:25.383412 2583 status_manager.go:244] "Starting to sync pod status with apiserver" Jul 13 22:13:25.424747 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:13:25.383451 2583 kubelet.go:2448] "Starting kubelet main sync loop" Jul 13 22:13:25.424747 ip-10-0-133-179 kubenswrapper[2583]: E0713 22:13:25.383493 2583 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful" Jul 13 22:13:25.424747 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:13:25.393458 2583 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160" Jul 13 22:13:25.584003 ip-10-0-133-179 kubenswrapper[2583]: E0713 22:13:25.474059 2583 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-133-179.ec2.internal\" not found" Jul 13 22:13:25.584003 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:13:25.484464 2583 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["kube-system/kube-apiserver-proxy-ip-10-0-133-179.ec2.internal","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-179.ec2.internal"] Jul 13 22:13:25.584003 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:13:25.484538 2583 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 13 22:13:25.584003 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:13:25.486238 2583 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-179.ec2.internal" event="NodeHasSufficientMemory" Jul 13 22:13:25.584003 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:13:25.486270 2583 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-179.ec2.internal" event="NodeHasNoDiskPressure" Jul 13 22:13:25.584003 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:13:25.486284 2583 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-179.ec2.internal" event="NodeHasSufficientPID" Jul 13 22:13:25.584003 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:13:25.490298 2583 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 13 22:13:25.584003 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:13:25.490414 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-133-179.ec2.internal" Jul 13 22:13:25.584003 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:13:25.490454 2583 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 13 22:13:25.584003 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:13:25.491109 2583 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-179.ec2.internal" event="NodeHasSufficientMemory" Jul 13 22:13:25.584003 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:13:25.491139 2583 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-179.ec2.internal" event="NodeHasNoDiskPressure" Jul 13 22:13:25.584003 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:13:25.491150 2583 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-179.ec2.internal" event="NodeHasSufficientPID" Jul 13 22:13:25.584003 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:13:25.491109 2583 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-179.ec2.internal" event="NodeHasSufficientMemory" Jul 13 22:13:25.584003 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:13:25.491208 2583 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-179.ec2.internal" event="NodeHasNoDiskPressure" Jul 13 22:13:25.584003 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:13:25.491218 2583 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-179.ec2.internal" event="NodeHasSufficientPID" Jul 13 22:13:25.584003 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:13:25.492294 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-179.ec2.internal" Jul 13 22:13:25.584003 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:13:25.492322 2583 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 13 22:13:25.584003 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:13:25.492993 2583 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-179.ec2.internal" event="NodeHasSufficientMemory" Jul 13 22:13:25.584003 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:13:25.493018 2583 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-179.ec2.internal" event="NodeHasNoDiskPressure" Jul 13 22:13:25.584003 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:13:25.493027 2583 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-179.ec2.internal" event="NodeHasSufficientPID" Jul 13 22:13:25.584003 ip-10-0-133-179 kubenswrapper[2583]: E0713 22:13:25.530309 2583 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-133-179.ec2.internal\" not found" node="ip-10-0-133-179.ec2.internal" Jul 13 22:13:25.584003 ip-10-0-133-179 kubenswrapper[2583]: E0713 22:13:25.535514 2583 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-133-179.ec2.internal\" not found" node="ip-10-0-133-179.ec2.internal" Jul 13 22:13:25.584003 ip-10-0-133-179 kubenswrapper[2583]: E0713 22:13:25.575060 2583 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-133-179.ec2.internal\" not found" Jul 13 22:13:25.584003 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:13:25.579425 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/120f6b7b6960a904bf802202dca0166e-config\") pod \"kube-apiserver-proxy-ip-10-0-133-179.ec2.internal\" (UID: \"120f6b7b6960a904bf802202dca0166e\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-133-179.ec2.internal" Jul 13 22:13:25.584003 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:13:25.579449 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/8bd03286da06035aeb817bb940fc3daa-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-133-179.ec2.internal\" (UID: \"8bd03286da06035aeb817bb940fc3daa\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-179.ec2.internal" Jul 13 22:13:25.584003 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:13:25.579475 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/8bd03286da06035aeb817bb940fc3daa-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-133-179.ec2.internal\" (UID: \"8bd03286da06035aeb817bb940fc3daa\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-179.ec2.internal" Jul 13 22:13:25.734919 ip-10-0-133-179 kubenswrapper[2583]: E0713 22:13:25.675636 2583 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-133-179.ec2.internal\" not found" Jul 13 22:13:25.734919 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:13:25.679838 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/120f6b7b6960a904bf802202dca0166e-config\") pod \"kube-apiserver-proxy-ip-10-0-133-179.ec2.internal\" (UID: \"120f6b7b6960a904bf802202dca0166e\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-133-179.ec2.internal" Jul 13 22:13:25.734919 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:13:25.679860 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/8bd03286da06035aeb817bb940fc3daa-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-133-179.ec2.internal\" (UID: \"8bd03286da06035aeb817bb940fc3daa\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-179.ec2.internal" Jul 13 22:13:25.734919 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:13:25.679878 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/8bd03286da06035aeb817bb940fc3daa-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-133-179.ec2.internal\" (UID: \"8bd03286da06035aeb817bb940fc3daa\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-179.ec2.internal" Jul 13 22:13:25.734919 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:13:25.679924 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/8bd03286da06035aeb817bb940fc3daa-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-133-179.ec2.internal\" (UID: \"8bd03286da06035aeb817bb940fc3daa\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-179.ec2.internal" Jul 13 22:13:25.734919 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:13:25.679942 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/8bd03286da06035aeb817bb940fc3daa-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-133-179.ec2.internal\" (UID: \"8bd03286da06035aeb817bb940fc3daa\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-179.ec2.internal" Jul 13 22:13:25.734919 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:13:25.679994 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/120f6b7b6960a904bf802202dca0166e-config\") pod \"kube-apiserver-proxy-ip-10-0-133-179.ec2.internal\" (UID: \"120f6b7b6960a904bf802202dca0166e\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-133-179.ec2.internal" Jul 13 22:13:25.862456 ip-10-0-133-179 kubenswrapper[2583]: E0713 22:13:25.776014 2583 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-133-179.ec2.internal\" not found" Jul 13 22:13:25.862456 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:13:25.832481 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-133-179.ec2.internal" Jul 13 22:13:25.862456 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:13:25.838049 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-179.ec2.internal" Jul 13 22:13:25.877803 ip-10-0-133-179 kubenswrapper[2583]: E0713 22:13:25.876147 2583 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-133-179.ec2.internal\" not found" Jul 13 22:13:26.024336 ip-10-0-133-179 kubenswrapper[2583]: E0713 22:13:25.976731 2583 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-133-179.ec2.internal\" not found" Jul 13 22:13:26.172547 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:13:26.071459 2583 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160" Jul 13 22:13:26.172547 ip-10-0-133-179 kubenswrapper[2583]: E0713 22:13:26.077006 2583 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-133-179.ec2.internal\" not found" Jul 13 22:13:26.172547 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:13:26.105276 2583 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials" Jul 13 22:13:26.172547 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:13:26.105472 2583 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received" Jul 13 22:13:26.172547 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:13:26.105492 2583 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received" Jul 13 22:13:26.272648 ip-10-0-133-179 kubenswrapper[2583]: E0713 22:13:26.177369 2583 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-133-179.ec2.internal\" not found" Jul 13 22:13:26.272648 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:13:26.177371 2583 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-07-12 22:08:25 +0000 UTC" deadline="2027-12-17 19:40:45.030826083 +0000 UTC" Jul 13 22:13:26.272648 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:13:26.177404 2583 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="12525h27m18.85342689s" Jul 13 22:13:26.272648 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:13:26.202680 2583 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160" Jul 13 22:13:26.277855 ip-10-0-133-179 kubenswrapper[2583]: E0713 22:13:26.277785 2583 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-133-179.ec2.internal\" not found" Jul 13 22:13:26.378239 ip-10-0-133-179 kubenswrapper[2583]: E0713 22:13:26.378200 2583 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-133-179.ec2.internal\" not found" Jul 13 22:13:26.408938 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:13:26.408914 2583 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160" Jul 13 22:13:26.476608 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:13:26.476563 2583 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-133-179.ec2.internal" Jul 13 22:13:26.486794 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:13:26.486772 2583 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Jul 13 22:13:26.487810 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:13:26.487792 2583 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-179.ec2.internal" Jul 13 22:13:26.498035 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:13:26.498016 2583 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Jul 13 22:13:26.538911 ip-10-0-133-179 kubenswrapper[2583]: W0713 22:13:26.538879 2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod120f6b7b6960a904bf802202dca0166e.slice/crio-231630cc5d3e097bd5095f34a342ca662a4bb12a3e0ee606a320bfe89bdfd86d WatchSource:0}: Error finding container 231630cc5d3e097bd5095f34a342ca662a4bb12a3e0ee606a320bfe89bdfd86d: Status 404 returned error can't find the container with id 231630cc5d3e097bd5095f34a342ca662a4bb12a3e0ee606a320bfe89bdfd86d Jul 13 22:13:26.539133 ip-10-0-133-179 kubenswrapper[2583]: W0713 22:13:26.539102 2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8bd03286da06035aeb817bb940fc3daa.slice/crio-e7a6588c4036cd616befd7cf65ee0b76f8cdd0533bbe11804c648fd2b6350201 WatchSource:0}: Error finding container e7a6588c4036cd616befd7cf65ee0b76f8cdd0533bbe11804c648fd2b6350201: Status 404 returned error can't find the container with id e7a6588c4036cd616befd7cf65ee0b76f8cdd0533bbe11804c648fd2b6350201 Jul 13 22:13:26.545340 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:13:26.545323 2583 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 13 22:13:26.910884 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:13:26.910801 2583 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160" Jul 13 22:13:27.149347 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:13:27.149319 2583 apiserver.go:52] "Watching apiserver" Jul 13 22:13:27.156174 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:13:27.156146 2583 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66" Jul 13 22:13:27.156639 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:13:27.156611 2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kube-system/konnectivity-agent-wgcj4","kube-system/kube-apiserver-proxy-ip-10-0-133-179.ec2.internal","openshift-image-registry/node-ca-r26w6","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-179.ec2.internal","openshift-multus/network-metrics-daemon-26w5h","openshift-ovn-kubernetes/ovnkube-node-khllb","kube-system/global-pull-secret-syncer-9hl5q","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2r94b","openshift-cluster-node-tuning-operator/tuned-sht7p","openshift-dns/node-resolver-grjf8","openshift-multus/multus-2jc7k","openshift-multus/multus-additional-cni-plugins-nbdsh","openshift-network-diagnostics/network-check-target-v8mkx","openshift-network-operator/iptables-alerter-brfkn"] Jul 13 22:13:27.158330 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:13:27.158310 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-grjf8" Jul 13 22:13:27.159713 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:13:27.159688 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-v8mkx" Jul 13 22:13:27.159803 ip-10-0-133-179 kubenswrapper[2583]: E0713 22:13:27.159769 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-v8mkx" podUID="c8bbc3a9-6b5c-485d-b7ef-ca4cdca462ce" Jul 13 22:13:27.160399 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:13:27.160381 2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\"" Jul 13 22:13:27.160484 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:13:27.160429 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-f69c6\"" Jul 13 22:13:27.160484 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:13:27.160432 2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\"" Jul 13 22:13:27.161253 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:13:27.161203 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-r26w6" Jul 13 22:13:27.162367 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:13:27.162345 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-brfkn" Jul 13 22:13:27.162975 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:13:27.162954 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-lxtpf\"" Jul 13 22:13:27.163126 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:13:27.163093 2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\"" Jul 13 22:13:27.163205 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:13:27.163190 2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\"" Jul 13 22:13:27.163268 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:13:27.163216 2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\"" Jul 13 22:13:27.163858 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:13:27.163620 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-26w5h" Jul 13 22:13:27.163858 ip-10-0-133-179 kubenswrapper[2583]: E0713 22:13:27.163690 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-26w5h" podUID="85265522-e114-4c04-b595-a26fcae86884" Jul 13 22:13:27.164729 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:13:27.164464 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-6vmvl\"" Jul 13 22:13:27.164729 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:13:27.164476 2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\"" Jul 13 22:13:27.164729 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:13:27.164485 2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\"" Jul 13 22:13:27.164898 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:13:27.164865 2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\"" Jul 13 22:13:27.164984 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:13:27.164959 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-khllb" Jul 13 22:13:27.167247 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:13:27.166517 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-sht7p" Jul 13 22:13:27.167247 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:13:27.166629 2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\"" Jul 13 22:13:27.167247 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:13:27.166952 2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\"" Jul 13 22:13:27.167247 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:13:27.167153 2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\"" Jul 13 22:13:27.167247 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:13:27.167166 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-ltgb7\"" Jul 13 22:13:27.167877 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:13:27.167422 2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\"" Jul 13 22:13:27.167877 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:13:27.167822 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-9hl5q" Jul 13 22:13:27.167978 ip-10-0-133-179 kubenswrapper[2583]: E0713 22:13:27.167887 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-9hl5q" podUID="d38c850d-d65b-4f99-ad24-5c1d443bff4b" Jul 13 22:13:27.168590 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:13:27.168414 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\"" Jul 13 22:13:27.168590 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:13:27.168441 2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\"" Jul 13 22:13:27.169143 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:13:27.169124 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2r94b" Jul 13 22:13:27.169348 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:13:27.169326 2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\"" Jul 13 22:13:27.169633 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:13:27.169554 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-6c546\"" Jul 13 22:13:27.169730 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:13:27.169714 2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\"" Jul 13 22:13:27.170419 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:13:27.170356 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-wgcj4" Jul 13 22:13:27.171012 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:13:27.170988 2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\"" Jul 13 22:13:27.172247 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:13:27.172037 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\"" Jul 13 22:13:27.172247 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:13:27.172119 2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\"" Jul 13 22:13:27.172247 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:13:27.172186 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-47lqv\"" Jul 13 22:13:27.173337 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:13:27.173103 2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\"" Jul 13 22:13:27.173337 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:13:27.173230 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-7fllz\"" Jul 13 22:13:27.173337 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:13:27.173313 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\"" Jul 13 22:13:27.174465 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:13:27.174399 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-2jc7k" Jul 13 22:13:27.177282 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:13:27.177262 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-nbdsh" Jul 13 22:13:27.177679 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:13:27.177652 2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\"" Jul 13 22:13:27.177790 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:13:27.177776 2583 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving" Jul 13 22:13:27.177973 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:13:27.177958 2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\"" Jul 13 22:13:27.178130 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:13:27.177977 2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\"" Jul 13 22:13:27.178198 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:13:27.178151 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-wxjkw\"" Jul 13 22:13:27.178249 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:13:27.178020 2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\"" Jul 13 22:13:27.179100 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:13:27.179079 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-f9dgc\"" Jul 13 22:13:27.179199 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:13:27.179079 2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\"" Jul 13 22:13:27.179901 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:13:27.179884 2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\"" Jul 13 22:13:27.186325 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:13:27.186288 2583 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Jul 13 22:13:27.186325 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:13:27.186310 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6fxhd\" (UniqueName: \"kubernetes.io/projected/03f58cdb-4d8b-41a9-aa82-dff50f158f0c-kube-api-access-6fxhd\") pod \"multus-additional-cni-plugins-nbdsh\" (UID: \"03f58cdb-4d8b-41a9-aa82-dff50f158f0c\") " pod="openshift-multus/multus-additional-cni-plugins-nbdsh" Jul 13 22:13:27.186501 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:13:27.186346 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tcrhd\" (UniqueName: \"kubernetes.io/projected/c8bbc3a9-6b5c-485d-b7ef-ca4cdca462ce-kube-api-access-tcrhd\") pod \"network-check-target-v8mkx\" (UID: \"c8bbc3a9-6b5c-485d-b7ef-ca4cdca462ce\") " pod="openshift-network-diagnostics/network-check-target-v8mkx" Jul 13 22:13:27.186501 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:13:27.186371 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xf67g\" (UniqueName: \"kubernetes.io/projected/3357149c-54b6-4e57-93f0-2b9bf43ea301-kube-api-access-xf67g\") pod \"iptables-alerter-brfkn\" (UID: \"3357149c-54b6-4e57-93f0-2b9bf43ea301\") " pod="openshift-network-operator/iptables-alerter-brfkn" Jul 13 22:13:27.186501 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:13:27.186397 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/19a018fb-4dd4-4a76-9305-8f70a789d38d-run-systemd\") pod \"ovnkube-node-khllb\" (UID: \"19a018fb-4dd4-4a76-9305-8f70a789d38d\") " pod="openshift-ovn-kubernetes/ovnkube-node-khllb" Jul 13 22:13:27.186501 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:13:27.186422 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/85265522-e114-4c04-b595-a26fcae86884-metrics-certs\") pod \"network-metrics-daemon-26w5h\" (UID: \"85265522-e114-4c04-b595-a26fcae86884\") " pod="openshift-multus/network-metrics-daemon-26w5h" Jul 13 22:13:27.186501 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:13:27.186442 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/19a018fb-4dd4-4a76-9305-8f70a789d38d-var-lib-openvswitch\") pod \"ovnkube-node-khllb\" (UID: \"19a018fb-4dd4-4a76-9305-8f70a789d38d\") " pod="openshift-ovn-kubernetes/ovnkube-node-khllb" Jul 13 22:13:27.186501 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:13:27.186465 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/19a018fb-4dd4-4a76-9305-8f70a789d38d-host-cni-netd\") pod \"ovnkube-node-khllb\" (UID: \"19a018fb-4dd4-4a76-9305-8f70a789d38d\") " pod="openshift-ovn-kubernetes/ovnkube-node-khllb" Jul 13 22:13:27.186501 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:13:27.186488 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/19a018fb-4dd4-4a76-9305-8f70a789d38d-ovn-node-metrics-cert\") pod \"ovnkube-node-khllb\" (UID: \"19a018fb-4dd4-4a76-9305-8f70a789d38d\") " pod="openshift-ovn-kubernetes/ovnkube-node-khllb" Jul 13 22:13:27.186804 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:13:27.186508 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/e05ae6ca-894a-4061-91ac-a0a6ed25d1b2-etc-sysctl-d\") pod \"tuned-sht7p\" (UID: \"e05ae6ca-894a-4061-91ac-a0a6ed25d1b2\") " pod="openshift-cluster-node-tuning-operator/tuned-sht7p" Jul 13 22:13:27.186804 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:13:27.186526 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/e05ae6ca-894a-4061-91ac-a0a6ed25d1b2-sys\") pod \"tuned-sht7p\" (UID: \"e05ae6ca-894a-4061-91ac-a0a6ed25d1b2\") " pod="openshift-cluster-node-tuning-operator/tuned-sht7p" Jul 13 22:13:27.186804 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:13:27.186590 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/01c7e40b-ef15-4cf6-b5e7-e7129825d668-host-run-k8s-cni-cncf-io\") pod \"multus-2jc7k\" (UID: \"01c7e40b-ef15-4cf6-b5e7-e7129825d668\") " pod="openshift-multus/multus-2jc7k" Jul 13 22:13:27.186804 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:13:27.186617 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/03f58cdb-4d8b-41a9-aa82-dff50f158f0c-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-nbdsh\" (UID: \"03f58cdb-4d8b-41a9-aa82-dff50f158f0c\") " pod="openshift-multus/multus-additional-cni-plugins-nbdsh" Jul 13 22:13:27.186804 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:13:27.186644 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/e05ae6ca-894a-4061-91ac-a0a6ed25d1b2-etc-sysctl-conf\") pod \"tuned-sht7p\" (UID: \"e05ae6ca-894a-4061-91ac-a0a6ed25d1b2\") " pod="openshift-cluster-node-tuning-operator/tuned-sht7p" Jul 13 22:13:27.186804 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:13:27.186665 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/e05ae6ca-894a-4061-91ac-a0a6ed25d1b2-tmp\") pod \"tuned-sht7p\" (UID: \"e05ae6ca-894a-4061-91ac-a0a6ed25d1b2\") " pod="openshift-cluster-node-tuning-operator/tuned-sht7p" Jul 13 22:13:27.186804 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:13:27.186682 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/ad16aa2f-aede-4d47-82d6-0ab7dda663d4-agent-certs\") pod \"konnectivity-agent-wgcj4\" (UID: \"ad16aa2f-aede-4d47-82d6-0ab7dda663d4\") " pod="kube-system/konnectivity-agent-wgcj4" Jul 13 22:13:27.186804 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:13:27.186698 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/19a018fb-4dd4-4a76-9305-8f70a789d38d-ovnkube-config\") pod \"ovnkube-node-khllb\" (UID: \"19a018fb-4dd4-4a76-9305-8f70a789d38d\") " pod="openshift-ovn-kubernetes/ovnkube-node-khllb" Jul 13 22:13:27.186804 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:13:27.186723 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-47txq\" (UniqueName: \"kubernetes.io/projected/19a018fb-4dd4-4a76-9305-8f70a789d38d-kube-api-access-47txq\") pod \"ovnkube-node-khllb\" (UID: \"19a018fb-4dd4-4a76-9305-8f70a789d38d\") " pod="openshift-ovn-kubernetes/ovnkube-node-khllb" Jul 13 22:13:27.186804 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:13:27.186738 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/e05ae6ca-894a-4061-91ac-a0a6ed25d1b2-etc-sysconfig\") pod \"tuned-sht7p\" (UID: \"e05ae6ca-894a-4061-91ac-a0a6ed25d1b2\") " pod="openshift-cluster-node-tuning-operator/tuned-sht7p" Jul 13 22:13:27.186804 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:13:27.186758 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w5f8s\" (UniqueName: \"kubernetes.io/projected/e05ae6ca-894a-4061-91ac-a0a6ed25d1b2-kube-api-access-w5f8s\") pod \"tuned-sht7p\" (UID: \"e05ae6ca-894a-4061-91ac-a0a6ed25d1b2\") " pod="openshift-cluster-node-tuning-operator/tuned-sht7p" Jul 13 22:13:27.186804 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:13:27.186783 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/5b796097-01b1-4a5d-a10e-2cf14087d82e-etc-selinux\") pod \"aws-ebs-csi-driver-node-2r94b\" (UID: \"5b796097-01b1-4a5d-a10e-2cf14087d82e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2r94b" Jul 13 22:13:27.186804 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:13:27.186806 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/01c7e40b-ef15-4cf6-b5e7-e7129825d668-host-var-lib-kubelet\") pod \"multus-2jc7k\" (UID: \"01c7e40b-ef15-4cf6-b5e7-e7129825d668\") " pod="openshift-multus/multus-2jc7k" Jul 13 22:13:27.187410 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:13:27.186849 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/01c7e40b-ef15-4cf6-b5e7-e7129825d668-cni-binary-copy\") pod \"multus-2jc7k\" (UID: \"01c7e40b-ef15-4cf6-b5e7-e7129825d668\") " pod="openshift-multus/multus-2jc7k" Jul 13 22:13:27.187410 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:13:27.186878 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/19a018fb-4dd4-4a76-9305-8f70a789d38d-systemd-units\") pod \"ovnkube-node-khllb\" (UID: \"19a018fb-4dd4-4a76-9305-8f70a789d38d\") " pod="openshift-ovn-kubernetes/ovnkube-node-khllb" Jul 13 22:13:27.187410 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:13:27.186896 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gtqs2\" (UniqueName: \"kubernetes.io/projected/5b796097-01b1-4a5d-a10e-2cf14087d82e-kube-api-access-gtqs2\") pod \"aws-ebs-csi-driver-node-2r94b\" (UID: \"5b796097-01b1-4a5d-a10e-2cf14087d82e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2r94b" Jul 13 22:13:27.187410 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:13:27.186910 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/01c7e40b-ef15-4cf6-b5e7-e7129825d668-multus-cni-dir\") pod \"multus-2jc7k\" (UID: \"01c7e40b-ef15-4cf6-b5e7-e7129825d668\") " pod="openshift-multus/multus-2jc7k" Jul 13 22:13:27.187410 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:13:27.186923 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sxh8f\" (UniqueName: \"kubernetes.io/projected/e8e90a01-fd99-4a55-ba17-522966f4fabe-kube-api-access-sxh8f\") pod \"node-resolver-grjf8\" (UID: \"e8e90a01-fd99-4a55-ba17-522966f4fabe\") " pod="openshift-dns/node-resolver-grjf8" Jul 13 22:13:27.187410 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:13:27.186969 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/19a018fb-4dd4-4a76-9305-8f70a789d38d-etc-openvswitch\") pod \"ovnkube-node-khllb\" (UID: \"19a018fb-4dd4-4a76-9305-8f70a789d38d\") " pod="openshift-ovn-kubernetes/ovnkube-node-khllb" Jul 13 22:13:27.187410 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:13:27.186993 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/e05ae6ca-894a-4061-91ac-a0a6ed25d1b2-etc-systemd\") pod \"tuned-sht7p\" (UID: \"e05ae6ca-894a-4061-91ac-a0a6ed25d1b2\") " pod="openshift-cluster-node-tuning-operator/tuned-sht7p" Jul 13 22:13:27.187410 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:13:27.187015 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/e05ae6ca-894a-4061-91ac-a0a6ed25d1b2-lib-modules\") pod \"tuned-sht7p\" (UID: \"e05ae6ca-894a-4061-91ac-a0a6ed25d1b2\") " pod="openshift-cluster-node-tuning-operator/tuned-sht7p" Jul 13 22:13:27.187410 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:13:27.187043 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-45p8v\" (UniqueName: \"kubernetes.io/projected/54e8db97-3d72-4aa1-80bb-e56f10c7e474-kube-api-access-45p8v\") pod \"node-ca-r26w6\" (UID: \"54e8db97-3d72-4aa1-80bb-e56f10c7e474\") " pod="openshift-image-registry/node-ca-r26w6" Jul 13 22:13:27.187410 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:13:27.187096 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/5b796097-01b1-4a5d-a10e-2cf14087d82e-registration-dir\") pod \"aws-ebs-csi-driver-node-2r94b\" (UID: \"5b796097-01b1-4a5d-a10e-2cf14087d82e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2r94b" Jul 13 22:13:27.187410 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:13:27.187124 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/01c7e40b-ef15-4cf6-b5e7-e7129825d668-host-var-lib-cni-bin\") pod \"multus-2jc7k\" (UID: \"01c7e40b-ef15-4cf6-b5e7-e7129825d668\") " pod="openshift-multus/multus-2jc7k" Jul 13 22:13:27.187410 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:13:27.187156 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/19a018fb-4dd4-4a76-9305-8f70a789d38d-env-overrides\") pod \"ovnkube-node-khllb\" (UID: \"19a018fb-4dd4-4a76-9305-8f70a789d38d\") " pod="openshift-ovn-kubernetes/ovnkube-node-khllb" Jul 13 22:13:27.187410 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:13:27.187180 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/d38c850d-d65b-4f99-ad24-5c1d443bff4b-kubelet-config\") pod \"global-pull-secret-syncer-9hl5q\" (UID: \"d38c850d-d65b-4f99-ad24-5c1d443bff4b\") " pod="kube-system/global-pull-secret-syncer-9hl5q" Jul 13 22:13:27.187410 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:13:27.187213 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/01c7e40b-ef15-4cf6-b5e7-e7129825d668-os-release\") pod \"multus-2jc7k\" (UID: \"01c7e40b-ef15-4cf6-b5e7-e7129825d668\") " pod="openshift-multus/multus-2jc7k" Jul 13 22:13:27.187410 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:13:27.187258 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/01c7e40b-ef15-4cf6-b5e7-e7129825d668-hostroot\") pod \"multus-2jc7k\" (UID: \"01c7e40b-ef15-4cf6-b5e7-e7129825d668\") " pod="openshift-multus/multus-2jc7k" Jul 13 22:13:27.187410 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:13:27.187283 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/03f58cdb-4d8b-41a9-aa82-dff50f158f0c-tuning-conf-dir\") pod \"multus-additional-cni-plugins-nbdsh\" (UID: \"03f58cdb-4d8b-41a9-aa82-dff50f158f0c\") " pod="openshift-multus/multus-additional-cni-plugins-nbdsh" Jul 13 22:13:27.188060 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:13:27.187311 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/3357149c-54b6-4e57-93f0-2b9bf43ea301-host-slash\") pod \"iptables-alerter-brfkn\" (UID: \"3357149c-54b6-4e57-93f0-2b9bf43ea301\") " pod="openshift-network-operator/iptables-alerter-brfkn" Jul 13 22:13:27.188060 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:13:27.187333 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/e05ae6ca-894a-4061-91ac-a0a6ed25d1b2-host\") pod \"tuned-sht7p\" (UID: \"e05ae6ca-894a-4061-91ac-a0a6ed25d1b2\") " pod="openshift-cluster-node-tuning-operator/tuned-sht7p" Jul 13 22:13:27.188060 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:13:27.187354 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/ad16aa2f-aede-4d47-82d6-0ab7dda663d4-konnectivity-ca\") pod \"konnectivity-agent-wgcj4\" (UID: \"ad16aa2f-aede-4d47-82d6-0ab7dda663d4\") " pod="kube-system/konnectivity-agent-wgcj4" Jul 13 22:13:27.188060 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:13:27.187368 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/19a018fb-4dd4-4a76-9305-8f70a789d38d-host-run-netns\") pod \"ovnkube-node-khllb\" (UID: \"19a018fb-4dd4-4a76-9305-8f70a789d38d\") " pod="openshift-ovn-kubernetes/ovnkube-node-khllb" Jul 13 22:13:27.188060 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:13:27.187381 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/19a018fb-4dd4-4a76-9305-8f70a789d38d-log-socket\") pod \"ovnkube-node-khllb\" (UID: \"19a018fb-4dd4-4a76-9305-8f70a789d38d\") " pod="openshift-ovn-kubernetes/ovnkube-node-khllb" Jul 13 22:13:27.188060 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:13:27.187400 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/e05ae6ca-894a-4061-91ac-a0a6ed25d1b2-etc-modprobe-d\") pod \"tuned-sht7p\" (UID: \"e05ae6ca-894a-4061-91ac-a0a6ed25d1b2\") " pod="openshift-cluster-node-tuning-operator/tuned-sht7p" Jul 13 22:13:27.188060 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:13:27.187416 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/e05ae6ca-894a-4061-91ac-a0a6ed25d1b2-etc-kubernetes\") pod \"tuned-sht7p\" (UID: \"e05ae6ca-894a-4061-91ac-a0a6ed25d1b2\") " pod="openshift-cluster-node-tuning-operator/tuned-sht7p" Jul 13 22:13:27.188060 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:13:27.187436 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/5b796097-01b1-4a5d-a10e-2cf14087d82e-device-dir\") pod \"aws-ebs-csi-driver-node-2r94b\" (UID: \"5b796097-01b1-4a5d-a10e-2cf14087d82e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2r94b" Jul 13 22:13:27.188060 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:13:27.187449 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/01c7e40b-ef15-4cf6-b5e7-e7129825d668-multus-conf-dir\") pod \"multus-2jc7k\" (UID: \"01c7e40b-ef15-4cf6-b5e7-e7129825d668\") " pod="openshift-multus/multus-2jc7k" Jul 13 22:13:27.188060 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:13:27.187466 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zwfk5\" (UniqueName: \"kubernetes.io/projected/85265522-e114-4c04-b595-a26fcae86884-kube-api-access-zwfk5\") pod \"network-metrics-daemon-26w5h\" (UID: \"85265522-e114-4c04-b595-a26fcae86884\") " pod="openshift-multus/network-metrics-daemon-26w5h" Jul 13 22:13:27.188060 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:13:27.187492 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/5b796097-01b1-4a5d-a10e-2cf14087d82e-kubelet-dir\") pod \"aws-ebs-csi-driver-node-2r94b\" (UID: \"5b796097-01b1-4a5d-a10e-2cf14087d82e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2r94b" Jul 13 22:13:27.188060 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:13:27.187514 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/01c7e40b-ef15-4cf6-b5e7-e7129825d668-multus-daemon-config\") pod \"multus-2jc7k\" (UID: \"01c7e40b-ef15-4cf6-b5e7-e7129825d668\") " pod="openshift-multus/multus-2jc7k" Jul 13 22:13:27.188060 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:13:27.187532 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/03f58cdb-4d8b-41a9-aa82-dff50f158f0c-os-release\") pod \"multus-additional-cni-plugins-nbdsh\" (UID: \"03f58cdb-4d8b-41a9-aa82-dff50f158f0c\") " pod="openshift-multus/multus-additional-cni-plugins-nbdsh" Jul 13 22:13:27.188060 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:13:27.187578 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/03f58cdb-4d8b-41a9-aa82-dff50f158f0c-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-nbdsh\" (UID: \"03f58cdb-4d8b-41a9-aa82-dff50f158f0c\") " pod="openshift-multus/multus-additional-cni-plugins-nbdsh" Jul 13 22:13:27.188060 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:13:27.187626 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/19a018fb-4dd4-4a76-9305-8f70a789d38d-host-kubelet\") pod \"ovnkube-node-khllb\" (UID: \"19a018fb-4dd4-4a76-9305-8f70a789d38d\") " pod="openshift-ovn-kubernetes/ovnkube-node-khllb" Jul 13 22:13:27.188060 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:13:27.187673 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/19a018fb-4dd4-4a76-9305-8f70a789d38d-host-cni-bin\") pod \"ovnkube-node-khllb\" (UID: \"19a018fb-4dd4-4a76-9305-8f70a789d38d\") " pod="openshift-ovn-kubernetes/ovnkube-node-khllb" Jul 13 22:13:27.188844 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:13:27.187699 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/19a018fb-4dd4-4a76-9305-8f70a789d38d-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-khllb\" (UID: \"19a018fb-4dd4-4a76-9305-8f70a789d38d\") " pod="openshift-ovn-kubernetes/ovnkube-node-khllb" Jul 13 22:13:27.188844 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:13:27.187726 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/e05ae6ca-894a-4061-91ac-a0a6ed25d1b2-etc-tuned\") pod \"tuned-sht7p\" (UID: \"e05ae6ca-894a-4061-91ac-a0a6ed25d1b2\") " pod="openshift-cluster-node-tuning-operator/tuned-sht7p" Jul 13 22:13:27.188844 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:13:27.187751 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/d38c850d-d65b-4f99-ad24-5c1d443bff4b-original-pull-secret\") pod \"global-pull-secret-syncer-9hl5q\" (UID: \"d38c850d-d65b-4f99-ad24-5c1d443bff4b\") " pod="kube-system/global-pull-secret-syncer-9hl5q" Jul 13 22:13:27.188844 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:13:27.187775 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/01c7e40b-ef15-4cf6-b5e7-e7129825d668-system-cni-dir\") pod \"multus-2jc7k\" (UID: \"01c7e40b-ef15-4cf6-b5e7-e7129825d668\") " pod="openshift-multus/multus-2jc7k" Jul 13 22:13:27.188844 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:13:27.187805 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/01c7e40b-ef15-4cf6-b5e7-e7129825d668-host-var-lib-cni-multus\") pod \"multus-2jc7k\" (UID: \"01c7e40b-ef15-4cf6-b5e7-e7129825d668\") " pod="openshift-multus/multus-2jc7k" Jul 13 22:13:27.188844 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:13:27.187831 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/03f58cdb-4d8b-41a9-aa82-dff50f158f0c-cni-binary-copy\") pod \"multus-additional-cni-plugins-nbdsh\" (UID: \"03f58cdb-4d8b-41a9-aa82-dff50f158f0c\") " pod="openshift-multus/multus-additional-cni-plugins-nbdsh" Jul 13 22:13:27.188844 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:13:27.187857 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/e8e90a01-fd99-4a55-ba17-522966f4fabe-hosts-file\") pod \"node-resolver-grjf8\" (UID: \"e8e90a01-fd99-4a55-ba17-522966f4fabe\") " pod="openshift-dns/node-resolver-grjf8" Jul 13 22:13:27.188844 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:13:27.187892 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/19a018fb-4dd4-4a76-9305-8f70a789d38d-run-openvswitch\") pod \"ovnkube-node-khllb\" (UID: \"19a018fb-4dd4-4a76-9305-8f70a789d38d\") " pod="openshift-ovn-kubernetes/ovnkube-node-khllb" Jul 13 22:13:27.188844 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:13:27.187910 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/19a018fb-4dd4-4a76-9305-8f70a789d38d-host-run-ovn-kubernetes\") pod \"ovnkube-node-khllb\" (UID: \"19a018fb-4dd4-4a76-9305-8f70a789d38d\") " pod="openshift-ovn-kubernetes/ovnkube-node-khllb" Jul 13 22:13:27.188844 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:13:27.187926 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/01c7e40b-ef15-4cf6-b5e7-e7129825d668-host-run-multus-certs\") pod \"multus-2jc7k\" (UID: \"01c7e40b-ef15-4cf6-b5e7-e7129825d668\") " pod="openshift-multus/multus-2jc7k" Jul 13 22:13:27.188844 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:13:27.187941 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/01c7e40b-ef15-4cf6-b5e7-e7129825d668-etc-kubernetes\") pod \"multus-2jc7k\" (UID: \"01c7e40b-ef15-4cf6-b5e7-e7129825d668\") " pod="openshift-multus/multus-2jc7k" Jul 13 22:13:27.188844 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:13:27.187970 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xkrxz\" (UniqueName: \"kubernetes.io/projected/01c7e40b-ef15-4cf6-b5e7-e7129825d668-kube-api-access-xkrxz\") pod \"multus-2jc7k\" (UID: \"01c7e40b-ef15-4cf6-b5e7-e7129825d668\") " pod="openshift-multus/multus-2jc7k" Jul 13 22:13:27.188844 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:13:27.187994 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/3357149c-54b6-4e57-93f0-2b9bf43ea301-iptables-alerter-script\") pod \"iptables-alerter-brfkn\" (UID: \"3357149c-54b6-4e57-93f0-2b9bf43ea301\") " pod="openshift-network-operator/iptables-alerter-brfkn" Jul 13 22:13:27.188844 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:13:27.188016 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/19a018fb-4dd4-4a76-9305-8f70a789d38d-run-ovn\") pod \"ovnkube-node-khllb\" (UID: \"19a018fb-4dd4-4a76-9305-8f70a789d38d\") " pod="openshift-ovn-kubernetes/ovnkube-node-khllb" Jul 13 22:13:27.188844 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:13:27.188039 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/e05ae6ca-894a-4061-91ac-a0a6ed25d1b2-run\") pod \"tuned-sht7p\" (UID: \"e05ae6ca-894a-4061-91ac-a0a6ed25d1b2\") " pod="openshift-cluster-node-tuning-operator/tuned-sht7p" Jul 13 22:13:27.188844 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:13:27.188064 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/54e8db97-3d72-4aa1-80bb-e56f10c7e474-host\") pod \"node-ca-r26w6\" (UID: \"54e8db97-3d72-4aa1-80bb-e56f10c7e474\") " pod="openshift-image-registry/node-ca-r26w6" Jul 13 22:13:27.189587 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:13:27.188083 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/54e8db97-3d72-4aa1-80bb-e56f10c7e474-serviceca\") pod \"node-ca-r26w6\" (UID: \"54e8db97-3d72-4aa1-80bb-e56f10c7e474\") " pod="openshift-image-registry/node-ca-r26w6" Jul 13 22:13:27.189587 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:13:27.188108 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/01c7e40b-ef15-4cf6-b5e7-e7129825d668-host-run-netns\") pod \"multus-2jc7k\" (UID: \"01c7e40b-ef15-4cf6-b5e7-e7129825d668\") " pod="openshift-multus/multus-2jc7k" Jul 13 22:13:27.189587 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:13:27.188131 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/19a018fb-4dd4-4a76-9305-8f70a789d38d-ovnkube-script-lib\") pod \"ovnkube-node-khllb\" (UID: \"19a018fb-4dd4-4a76-9305-8f70a789d38d\") " pod="openshift-ovn-kubernetes/ovnkube-node-khllb" Jul 13 22:13:27.189587 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:13:27.188153 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/d38c850d-d65b-4f99-ad24-5c1d443bff4b-dbus\") pod \"global-pull-secret-syncer-9hl5q\" (UID: \"d38c850d-d65b-4f99-ad24-5c1d443bff4b\") " pod="kube-system/global-pull-secret-syncer-9hl5q" Jul 13 22:13:27.189587 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:13:27.188176 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/5b796097-01b1-4a5d-a10e-2cf14087d82e-socket-dir\") pod \"aws-ebs-csi-driver-node-2r94b\" (UID: \"5b796097-01b1-4a5d-a10e-2cf14087d82e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2r94b" Jul 13 22:13:27.189587 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:13:27.188201 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/03f58cdb-4d8b-41a9-aa82-dff50f158f0c-system-cni-dir\") pod \"multus-additional-cni-plugins-nbdsh\" (UID: \"03f58cdb-4d8b-41a9-aa82-dff50f158f0c\") " pod="openshift-multus/multus-additional-cni-plugins-nbdsh" Jul 13 22:13:27.189587 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:13:27.188221 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e8e90a01-fd99-4a55-ba17-522966f4fabe-tmp-dir\") pod \"node-resolver-grjf8\" (UID: \"e8e90a01-fd99-4a55-ba17-522966f4fabe\") " pod="openshift-dns/node-resolver-grjf8" Jul 13 22:13:27.189587 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:13:27.188236 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/19a018fb-4dd4-4a76-9305-8f70a789d38d-host-slash\") pod \"ovnkube-node-khllb\" (UID: \"19a018fb-4dd4-4a76-9305-8f70a789d38d\") " pod="openshift-ovn-kubernetes/ovnkube-node-khllb" Jul 13 22:13:27.189587 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:13:27.188249 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/19a018fb-4dd4-4a76-9305-8f70a789d38d-node-log\") pod \"ovnkube-node-khllb\" (UID: \"19a018fb-4dd4-4a76-9305-8f70a789d38d\") " pod="openshift-ovn-kubernetes/ovnkube-node-khllb" Jul 13 22:13:27.189587 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:13:27.188274 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/e05ae6ca-894a-4061-91ac-a0a6ed25d1b2-var-lib-kubelet\") pod \"tuned-sht7p\" (UID: \"e05ae6ca-894a-4061-91ac-a0a6ed25d1b2\") " pod="openshift-cluster-node-tuning-operator/tuned-sht7p" Jul 13 22:13:27.189587 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:13:27.188295 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/5b796097-01b1-4a5d-a10e-2cf14087d82e-sys-fs\") pod \"aws-ebs-csi-driver-node-2r94b\" (UID: \"5b796097-01b1-4a5d-a10e-2cf14087d82e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2r94b" Jul 13 22:13:27.189587 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:13:27.188314 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/01c7e40b-ef15-4cf6-b5e7-e7129825d668-cnibin\") pod \"multus-2jc7k\" (UID: \"01c7e40b-ef15-4cf6-b5e7-e7129825d668\") " pod="openshift-multus/multus-2jc7k" Jul 13 22:13:27.189587 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:13:27.188343 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/01c7e40b-ef15-4cf6-b5e7-e7129825d668-multus-socket-dir-parent\") pod \"multus-2jc7k\" (UID: \"01c7e40b-ef15-4cf6-b5e7-e7129825d668\") " pod="openshift-multus/multus-2jc7k" Jul 13 22:13:27.189587 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:13:27.188371 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/03f58cdb-4d8b-41a9-aa82-dff50f158f0c-cnibin\") pod \"multus-additional-cni-plugins-nbdsh\" (UID: \"03f58cdb-4d8b-41a9-aa82-dff50f158f0c\") " pod="openshift-multus/multus-additional-cni-plugins-nbdsh" Jul 13 22:13:27.217396 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:13:27.217370 2583 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-dc4tm" Jul 13 22:13:27.223364 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:13:27.223340 2583 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-dc4tm" Jul 13 22:13:27.278524 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:13:27.278463 2583 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world" Jul 13 22:13:27.288893 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:13:27.288857 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/01c7e40b-ef15-4cf6-b5e7-e7129825d668-os-release\") pod \"multus-2jc7k\" (UID: \"01c7e40b-ef15-4cf6-b5e7-e7129825d668\") " pod="openshift-multus/multus-2jc7k" Jul 13 22:13:27.289077 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:13:27.288903 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/01c7e40b-ef15-4cf6-b5e7-e7129825d668-hostroot\") pod \"multus-2jc7k\" (UID: \"01c7e40b-ef15-4cf6-b5e7-e7129825d668\") " pod="openshift-multus/multus-2jc7k" Jul 13 22:13:27.289077 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:13:27.288939 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/03f58cdb-4d8b-41a9-aa82-dff50f158f0c-tuning-conf-dir\") pod \"multus-additional-cni-plugins-nbdsh\" (UID: \"03f58cdb-4d8b-41a9-aa82-dff50f158f0c\") " pod="openshift-multus/multus-additional-cni-plugins-nbdsh" Jul 13 22:13:27.289077 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:13:27.288965 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/3357149c-54b6-4e57-93f0-2b9bf43ea301-host-slash\") pod \"iptables-alerter-brfkn\" (UID: \"3357149c-54b6-4e57-93f0-2b9bf43ea301\") " pod="openshift-network-operator/iptables-alerter-brfkn" Jul 13 22:13:27.289077 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:13:27.288990 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/e05ae6ca-894a-4061-91ac-a0a6ed25d1b2-host\") pod \"tuned-sht7p\" (UID: \"e05ae6ca-894a-4061-91ac-a0a6ed25d1b2\") " pod="openshift-cluster-node-tuning-operator/tuned-sht7p" Jul 13 22:13:27.289077 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:13:27.289011 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/ad16aa2f-aede-4d47-82d6-0ab7dda663d4-konnectivity-ca\") pod \"konnectivity-agent-wgcj4\" (UID: \"ad16aa2f-aede-4d47-82d6-0ab7dda663d4\") " pod="kube-system/konnectivity-agent-wgcj4" Jul 13 22:13:27.289077 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:13:27.289040 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/19a018fb-4dd4-4a76-9305-8f70a789d38d-host-run-netns\") pod \"ovnkube-node-khllb\" (UID: \"19a018fb-4dd4-4a76-9305-8f70a789d38d\") " pod="openshift-ovn-kubernetes/ovnkube-node-khllb" Jul 13 22:13:27.289077 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:13:27.289063 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/19a018fb-4dd4-4a76-9305-8f70a789d38d-log-socket\") pod \"ovnkube-node-khllb\" (UID: \"19a018fb-4dd4-4a76-9305-8f70a789d38d\") " pod="openshift-ovn-kubernetes/ovnkube-node-khllb" Jul 13 22:13:27.289415 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:13:27.289089 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/e05ae6ca-894a-4061-91ac-a0a6ed25d1b2-etc-modprobe-d\") pod \"tuned-sht7p\" (UID: \"e05ae6ca-894a-4061-91ac-a0a6ed25d1b2\") " pod="openshift-cluster-node-tuning-operator/tuned-sht7p" Jul 13 22:13:27.289415 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:13:27.289111 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/e05ae6ca-894a-4061-91ac-a0a6ed25d1b2-etc-kubernetes\") pod \"tuned-sht7p\" (UID: \"e05ae6ca-894a-4061-91ac-a0a6ed25d1b2\") " pod="openshift-cluster-node-tuning-operator/tuned-sht7p" Jul 13 22:13:27.289415 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:13:27.289135 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/5b796097-01b1-4a5d-a10e-2cf14087d82e-device-dir\") pod \"aws-ebs-csi-driver-node-2r94b\" (UID: \"5b796097-01b1-4a5d-a10e-2cf14087d82e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2r94b" Jul 13 22:13:27.289415 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:13:27.289157 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/01c7e40b-ef15-4cf6-b5e7-e7129825d668-multus-conf-dir\") pod \"multus-2jc7k\" (UID: \"01c7e40b-ef15-4cf6-b5e7-e7129825d668\") " pod="openshift-multus/multus-2jc7k" Jul 13 22:13:27.289415 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:13:27.289182 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zwfk5\" (UniqueName: \"kubernetes.io/projected/85265522-e114-4c04-b595-a26fcae86884-kube-api-access-zwfk5\") pod \"network-metrics-daemon-26w5h\" (UID: \"85265522-e114-4c04-b595-a26fcae86884\") " pod="openshift-multus/network-metrics-daemon-26w5h" Jul 13 22:13:27.289415 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:13:27.289204 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/5b796097-01b1-4a5d-a10e-2cf14087d82e-kubelet-dir\") pod \"aws-ebs-csi-driver-node-2r94b\" (UID: \"5b796097-01b1-4a5d-a10e-2cf14087d82e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2r94b" Jul 13 22:13:27.289415 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:13:27.289253 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/01c7e40b-ef15-4cf6-b5e7-e7129825d668-multus-daemon-config\") pod \"multus-2jc7k\" (UID: \"01c7e40b-ef15-4cf6-b5e7-e7129825d668\") " pod="openshift-multus/multus-2jc7k" Jul 13 22:13:27.289415 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:13:27.289283 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/03f58cdb-4d8b-41a9-aa82-dff50f158f0c-os-release\") pod \"multus-additional-cni-plugins-nbdsh\" (UID: \"03f58cdb-4d8b-41a9-aa82-dff50f158f0c\") " pod="openshift-multus/multus-additional-cni-plugins-nbdsh" Jul 13 22:13:27.289415 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:13:27.289307 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/03f58cdb-4d8b-41a9-aa82-dff50f158f0c-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-nbdsh\" (UID: \"03f58cdb-4d8b-41a9-aa82-dff50f158f0c\") " pod="openshift-multus/multus-additional-cni-plugins-nbdsh" Jul 13 22:13:27.289415 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:13:27.289333 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/19a018fb-4dd4-4a76-9305-8f70a789d38d-host-kubelet\") pod \"ovnkube-node-khllb\" (UID: \"19a018fb-4dd4-4a76-9305-8f70a789d38d\") " pod="openshift-ovn-kubernetes/ovnkube-node-khllb" Jul 13 22:13:27.289415 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:13:27.289375 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/19a018fb-4dd4-4a76-9305-8f70a789d38d-host-cni-bin\") pod \"ovnkube-node-khllb\" (UID: \"19a018fb-4dd4-4a76-9305-8f70a789d38d\") " pod="openshift-ovn-kubernetes/ovnkube-node-khllb" Jul 13 22:13:27.289415 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:13:27.289402 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/19a018fb-4dd4-4a76-9305-8f70a789d38d-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-khllb\" (UID: \"19a018fb-4dd4-4a76-9305-8f70a789d38d\") " pod="openshift-ovn-kubernetes/ovnkube-node-khllb" Jul 13 22:13:27.290003 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:13:27.289430 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/e05ae6ca-894a-4061-91ac-a0a6ed25d1b2-etc-tuned\") pod \"tuned-sht7p\" (UID: \"e05ae6ca-894a-4061-91ac-a0a6ed25d1b2\") " pod="openshift-cluster-node-tuning-operator/tuned-sht7p" Jul 13 22:13:27.290003 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:13:27.289456 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/d38c850d-d65b-4f99-ad24-5c1d443bff4b-original-pull-secret\") pod \"global-pull-secret-syncer-9hl5q\" (UID: \"d38c850d-d65b-4f99-ad24-5c1d443bff4b\") " pod="kube-system/global-pull-secret-syncer-9hl5q" Jul 13 22:13:27.290003 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:13:27.289478 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/01c7e40b-ef15-4cf6-b5e7-e7129825d668-system-cni-dir\") pod \"multus-2jc7k\" (UID: \"01c7e40b-ef15-4cf6-b5e7-e7129825d668\") " pod="openshift-multus/multus-2jc7k" Jul 13 22:13:27.290003 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:13:27.289517 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/01c7e40b-ef15-4cf6-b5e7-e7129825d668-host-var-lib-cni-multus\") pod \"multus-2jc7k\" (UID: \"01c7e40b-ef15-4cf6-b5e7-e7129825d668\") " pod="openshift-multus/multus-2jc7k" Jul 13 22:13:27.290003 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:13:27.289541 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/03f58cdb-4d8b-41a9-aa82-dff50f158f0c-cni-binary-copy\") pod \"multus-additional-cni-plugins-nbdsh\" (UID: \"03f58cdb-4d8b-41a9-aa82-dff50f158f0c\") " pod="openshift-multus/multus-additional-cni-plugins-nbdsh" Jul 13 22:13:27.290003 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:13:27.289564 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/e8e90a01-fd99-4a55-ba17-522966f4fabe-hosts-file\") pod \"node-resolver-grjf8\" (UID: \"e8e90a01-fd99-4a55-ba17-522966f4fabe\") " pod="openshift-dns/node-resolver-grjf8" Jul 13 22:13:27.290003 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:13:27.289608 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/19a018fb-4dd4-4a76-9305-8f70a789d38d-run-openvswitch\") pod \"ovnkube-node-khllb\" (UID: \"19a018fb-4dd4-4a76-9305-8f70a789d38d\") " pod="openshift-ovn-kubernetes/ovnkube-node-khllb" Jul 13 22:13:27.290003 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:13:27.289612 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/01c7e40b-ef15-4cf6-b5e7-e7129825d668-multus-conf-dir\") pod \"multus-2jc7k\" (UID: \"01c7e40b-ef15-4cf6-b5e7-e7129825d668\") " pod="openshift-multus/multus-2jc7k" Jul 13 22:13:27.290003 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:13:27.289634 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/19a018fb-4dd4-4a76-9305-8f70a789d38d-host-run-ovn-kubernetes\") pod \"ovnkube-node-khllb\" (UID: \"19a018fb-4dd4-4a76-9305-8f70a789d38d\") " pod="openshift-ovn-kubernetes/ovnkube-node-khllb" Jul 13 22:13:27.290003 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:13:27.289668 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/01c7e40b-ef15-4cf6-b5e7-e7129825d668-host-run-multus-certs\") pod \"multus-2jc7k\" (UID: \"01c7e40b-ef15-4cf6-b5e7-e7129825d668\") " pod="openshift-multus/multus-2jc7k" Jul 13 22:13:27.290003 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:13:27.289693 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/01c7e40b-ef15-4cf6-b5e7-e7129825d668-etc-kubernetes\") pod \"multus-2jc7k\" (UID: \"01c7e40b-ef15-4cf6-b5e7-e7129825d668\") " pod="openshift-multus/multus-2jc7k" Jul 13 22:13:27.290003 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:13:27.289699 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/01c7e40b-ef15-4cf6-b5e7-e7129825d668-os-release\") pod \"multus-2jc7k\" (UID: \"01c7e40b-ef15-4cf6-b5e7-e7129825d668\") " pod="openshift-multus/multus-2jc7k" Jul 13 22:13:27.290003 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:13:27.289723 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xkrxz\" (UniqueName: \"kubernetes.io/projected/01c7e40b-ef15-4cf6-b5e7-e7129825d668-kube-api-access-xkrxz\") pod \"multus-2jc7k\" (UID: \"01c7e40b-ef15-4cf6-b5e7-e7129825d668\") " pod="openshift-multus/multus-2jc7k" Jul 13 22:13:27.290003 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:13:27.289740 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/01c7e40b-ef15-4cf6-b5e7-e7129825d668-hostroot\") pod \"multus-2jc7k\" (UID: \"01c7e40b-ef15-4cf6-b5e7-e7129825d668\") " pod="openshift-multus/multus-2jc7k" Jul 13 22:13:27.290003 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:13:27.289749 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/3357149c-54b6-4e57-93f0-2b9bf43ea301-iptables-alerter-script\") pod \"iptables-alerter-brfkn\" (UID: \"3357149c-54b6-4e57-93f0-2b9bf43ea301\") " pod="openshift-network-operator/iptables-alerter-brfkn" Jul 13 22:13:27.290003 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:13:27.289771 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/19a018fb-4dd4-4a76-9305-8f70a789d38d-run-ovn\") pod \"ovnkube-node-khllb\" (UID: \"19a018fb-4dd4-4a76-9305-8f70a789d38d\") " pod="openshift-ovn-kubernetes/ovnkube-node-khllb" Jul 13 22:13:27.290003 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:13:27.289793 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/e05ae6ca-894a-4061-91ac-a0a6ed25d1b2-run\") pod \"tuned-sht7p\" (UID: \"e05ae6ca-894a-4061-91ac-a0a6ed25d1b2\") " pod="openshift-cluster-node-tuning-operator/tuned-sht7p" Jul 13 22:13:27.290003 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:13:27.289812 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/54e8db97-3d72-4aa1-80bb-e56f10c7e474-host\") pod \"node-ca-r26w6\" (UID: \"54e8db97-3d72-4aa1-80bb-e56f10c7e474\") " pod="openshift-image-registry/node-ca-r26w6" Jul 13 22:13:27.290824 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:13:27.289833 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/54e8db97-3d72-4aa1-80bb-e56f10c7e474-serviceca\") pod \"node-ca-r26w6\" (UID: \"54e8db97-3d72-4aa1-80bb-e56f10c7e474\") " pod="openshift-image-registry/node-ca-r26w6" Jul 13 22:13:27.290824 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:13:27.289873 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/01c7e40b-ef15-4cf6-b5e7-e7129825d668-host-run-netns\") pod \"multus-2jc7k\" (UID: \"01c7e40b-ef15-4cf6-b5e7-e7129825d668\") " pod="openshift-multus/multus-2jc7k" Jul 13 22:13:27.290824 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:13:27.289901 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/19a018fb-4dd4-4a76-9305-8f70a789d38d-ovnkube-script-lib\") pod \"ovnkube-node-khllb\" (UID: \"19a018fb-4dd4-4a76-9305-8f70a789d38d\") " pod="openshift-ovn-kubernetes/ovnkube-node-khllb" Jul 13 22:13:27.290824 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:13:27.289905 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/03f58cdb-4d8b-41a9-aa82-dff50f158f0c-tuning-conf-dir\") pod \"multus-additional-cni-plugins-nbdsh\" (UID: \"03f58cdb-4d8b-41a9-aa82-dff50f158f0c\") " pod="openshift-multus/multus-additional-cni-plugins-nbdsh" Jul 13 22:13:27.290824 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:13:27.289939 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/d38c850d-d65b-4f99-ad24-5c1d443bff4b-dbus\") pod \"global-pull-secret-syncer-9hl5q\" (UID: \"d38c850d-d65b-4f99-ad24-5c1d443bff4b\") " pod="kube-system/global-pull-secret-syncer-9hl5q" Jul 13 22:13:27.290824 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:13:27.289964 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/5b796097-01b1-4a5d-a10e-2cf14087d82e-socket-dir\") pod \"aws-ebs-csi-driver-node-2r94b\" (UID: \"5b796097-01b1-4a5d-a10e-2cf14087d82e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2r94b" Jul 13 22:13:27.290824 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:13:27.289988 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/03f58cdb-4d8b-41a9-aa82-dff50f158f0c-system-cni-dir\") pod \"multus-additional-cni-plugins-nbdsh\" (UID: \"03f58cdb-4d8b-41a9-aa82-dff50f158f0c\") " pod="openshift-multus/multus-additional-cni-plugins-nbdsh" Jul 13 22:13:27.290824 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:13:27.290011 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e8e90a01-fd99-4a55-ba17-522966f4fabe-tmp-dir\") pod \"node-resolver-grjf8\" (UID: \"e8e90a01-fd99-4a55-ba17-522966f4fabe\") " pod="openshift-dns/node-resolver-grjf8" Jul 13 22:13:27.290824 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:13:27.290036 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/19a018fb-4dd4-4a76-9305-8f70a789d38d-host-slash\") pod \"ovnkube-node-khllb\" (UID: \"19a018fb-4dd4-4a76-9305-8f70a789d38d\") " pod="openshift-ovn-kubernetes/ovnkube-node-khllb" Jul 13 22:13:27.290824 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:13:27.290060 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/19a018fb-4dd4-4a76-9305-8f70a789d38d-node-log\") pod \"ovnkube-node-khllb\" (UID: \"19a018fb-4dd4-4a76-9305-8f70a789d38d\") " pod="openshift-ovn-kubernetes/ovnkube-node-khllb" Jul 13 22:13:27.290824 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:13:27.290082 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/e05ae6ca-894a-4061-91ac-a0a6ed25d1b2-var-lib-kubelet\") pod \"tuned-sht7p\" (UID: \"e05ae6ca-894a-4061-91ac-a0a6ed25d1b2\") " pod="openshift-cluster-node-tuning-operator/tuned-sht7p" Jul 13 22:13:27.290824 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:13:27.290108 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/5b796097-01b1-4a5d-a10e-2cf14087d82e-sys-fs\") pod \"aws-ebs-csi-driver-node-2r94b\" (UID: \"5b796097-01b1-4a5d-a10e-2cf14087d82e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2r94b" Jul 13 22:13:27.290824 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:13:27.290132 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/01c7e40b-ef15-4cf6-b5e7-e7129825d668-cnibin\") pod \"multus-2jc7k\" (UID: \"01c7e40b-ef15-4cf6-b5e7-e7129825d668\") " pod="openshift-multus/multus-2jc7k" Jul 13 22:13:27.290824 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:13:27.290155 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/01c7e40b-ef15-4cf6-b5e7-e7129825d668-multus-socket-dir-parent\") pod \"multus-2jc7k\" (UID: \"01c7e40b-ef15-4cf6-b5e7-e7129825d668\") " pod="openshift-multus/multus-2jc7k" Jul 13 22:13:27.290824 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:13:27.290179 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/03f58cdb-4d8b-41a9-aa82-dff50f158f0c-cnibin\") pod \"multus-additional-cni-plugins-nbdsh\" (UID: \"03f58cdb-4d8b-41a9-aa82-dff50f158f0c\") " pod="openshift-multus/multus-additional-cni-plugins-nbdsh" Jul 13 22:13:27.290824 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:13:27.290213 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6fxhd\" (UniqueName: \"kubernetes.io/projected/03f58cdb-4d8b-41a9-aa82-dff50f158f0c-kube-api-access-6fxhd\") pod \"multus-additional-cni-plugins-nbdsh\" (UID: \"03f58cdb-4d8b-41a9-aa82-dff50f158f0c\") " pod="openshift-multus/multus-additional-cni-plugins-nbdsh" Jul 13 22:13:27.290824 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:13:27.290238 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tcrhd\" (UniqueName: \"kubernetes.io/projected/c8bbc3a9-6b5c-485d-b7ef-ca4cdca462ce-kube-api-access-tcrhd\") pod \"network-check-target-v8mkx\" (UID: \"c8bbc3a9-6b5c-485d-b7ef-ca4cdca462ce\") " pod="openshift-network-diagnostics/network-check-target-v8mkx" Jul 13 22:13:27.291558 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:13:27.290263 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xf67g\" (UniqueName: \"kubernetes.io/projected/3357149c-54b6-4e57-93f0-2b9bf43ea301-kube-api-access-xf67g\") pod \"iptables-alerter-brfkn\" (UID: \"3357149c-54b6-4e57-93f0-2b9bf43ea301\") " pod="openshift-network-operator/iptables-alerter-brfkn" Jul 13 22:13:27.291558 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:13:27.290285 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/19a018fb-4dd4-4a76-9305-8f70a789d38d-run-systemd\") pod \"ovnkube-node-khllb\" (UID: \"19a018fb-4dd4-4a76-9305-8f70a789d38d\") " pod="openshift-ovn-kubernetes/ovnkube-node-khllb" Jul 13 22:13:27.291558 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:13:27.290309 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/85265522-e114-4c04-b595-a26fcae86884-metrics-certs\") pod \"network-metrics-daemon-26w5h\" (UID: \"85265522-e114-4c04-b595-a26fcae86884\") " pod="openshift-multus/network-metrics-daemon-26w5h" Jul 13 22:13:27.291558 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:13:27.290334 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/19a018fb-4dd4-4a76-9305-8f70a789d38d-var-lib-openvswitch\") pod \"ovnkube-node-khllb\" (UID: \"19a018fb-4dd4-4a76-9305-8f70a789d38d\") " pod="openshift-ovn-kubernetes/ovnkube-node-khllb" Jul 13 22:13:27.291558 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:13:27.290359 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/19a018fb-4dd4-4a76-9305-8f70a789d38d-host-cni-netd\") pod \"ovnkube-node-khllb\" (UID: \"19a018fb-4dd4-4a76-9305-8f70a789d38d\") " pod="openshift-ovn-kubernetes/ovnkube-node-khllb" Jul 13 22:13:27.291558 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:13:27.290384 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/19a018fb-4dd4-4a76-9305-8f70a789d38d-ovn-node-metrics-cert\") pod \"ovnkube-node-khllb\" (UID: \"19a018fb-4dd4-4a76-9305-8f70a789d38d\") " pod="openshift-ovn-kubernetes/ovnkube-node-khllb" Jul 13 22:13:27.291558 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:13:27.290408 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/e05ae6ca-894a-4061-91ac-a0a6ed25d1b2-etc-sysctl-d\") pod \"tuned-sht7p\" (UID: \"e05ae6ca-894a-4061-91ac-a0a6ed25d1b2\") " pod="openshift-cluster-node-tuning-operator/tuned-sht7p" Jul 13 22:13:27.291558 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:13:27.290432 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/e05ae6ca-894a-4061-91ac-a0a6ed25d1b2-sys\") pod \"tuned-sht7p\" (UID: \"e05ae6ca-894a-4061-91ac-a0a6ed25d1b2\") " pod="openshift-cluster-node-tuning-operator/tuned-sht7p" Jul 13 22:13:27.291558 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:13:27.290458 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/01c7e40b-ef15-4cf6-b5e7-e7129825d668-host-run-k8s-cni-cncf-io\") pod \"multus-2jc7k\" (UID: \"01c7e40b-ef15-4cf6-b5e7-e7129825d668\") " pod="openshift-multus/multus-2jc7k" Jul 13 22:13:27.291558 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:13:27.290485 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/03f58cdb-4d8b-41a9-aa82-dff50f158f0c-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-nbdsh\" (UID: \"03f58cdb-4d8b-41a9-aa82-dff50f158f0c\") " pod="openshift-multus/multus-additional-cni-plugins-nbdsh" Jul 13 22:13:27.291558 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:13:27.290494 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/19a018fb-4dd4-4a76-9305-8f70a789d38d-ovnkube-script-lib\") pod \"ovnkube-node-khllb\" (UID: \"19a018fb-4dd4-4a76-9305-8f70a789d38d\") " pod="openshift-ovn-kubernetes/ovnkube-node-khllb" Jul 13 22:13:27.291558 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:13:27.290512 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/e05ae6ca-894a-4061-91ac-a0a6ed25d1b2-etc-sysctl-conf\") pod \"tuned-sht7p\" (UID: \"e05ae6ca-894a-4061-91ac-a0a6ed25d1b2\") " pod="openshift-cluster-node-tuning-operator/tuned-sht7p" Jul 13 22:13:27.291558 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:13:27.290536 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/e05ae6ca-894a-4061-91ac-a0a6ed25d1b2-tmp\") pod \"tuned-sht7p\" (UID: \"e05ae6ca-894a-4061-91ac-a0a6ed25d1b2\") " pod="openshift-cluster-node-tuning-operator/tuned-sht7p" Jul 13 22:13:27.291558 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:13:27.290560 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/ad16aa2f-aede-4d47-82d6-0ab7dda663d4-agent-certs\") pod \"konnectivity-agent-wgcj4\" (UID: \"ad16aa2f-aede-4d47-82d6-0ab7dda663d4\") " pod="kube-system/konnectivity-agent-wgcj4" Jul 13 22:13:27.291558 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:13:27.290603 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/19a018fb-4dd4-4a76-9305-8f70a789d38d-ovnkube-config\") pod \"ovnkube-node-khllb\" (UID: \"19a018fb-4dd4-4a76-9305-8f70a789d38d\") " pod="openshift-ovn-kubernetes/ovnkube-node-khllb" Jul 13 22:13:27.291558 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:13:27.290628 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-47txq\" (UniqueName: \"kubernetes.io/projected/19a018fb-4dd4-4a76-9305-8f70a789d38d-kube-api-access-47txq\") pod \"ovnkube-node-khllb\" (UID: \"19a018fb-4dd4-4a76-9305-8f70a789d38d\") " pod="openshift-ovn-kubernetes/ovnkube-node-khllb" Jul 13 22:13:27.291558 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:13:27.290652 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/e05ae6ca-894a-4061-91ac-a0a6ed25d1b2-etc-sysconfig\") pod \"tuned-sht7p\" (UID: \"e05ae6ca-894a-4061-91ac-a0a6ed25d1b2\") " pod="openshift-cluster-node-tuning-operator/tuned-sht7p" Jul 13 22:13:27.292344 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:13:27.290675 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w5f8s\" (UniqueName: \"kubernetes.io/projected/e05ae6ca-894a-4061-91ac-a0a6ed25d1b2-kube-api-access-w5f8s\") pod \"tuned-sht7p\" (UID: \"e05ae6ca-894a-4061-91ac-a0a6ed25d1b2\") " pod="openshift-cluster-node-tuning-operator/tuned-sht7p" Jul 13 22:13:27.292344 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:13:27.290702 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/5b796097-01b1-4a5d-a10e-2cf14087d82e-etc-selinux\") pod \"aws-ebs-csi-driver-node-2r94b\" (UID: \"5b796097-01b1-4a5d-a10e-2cf14087d82e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2r94b" Jul 13 22:13:27.292344 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:13:27.290727 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/01c7e40b-ef15-4cf6-b5e7-e7129825d668-host-var-lib-kubelet\") pod \"multus-2jc7k\" (UID: \"01c7e40b-ef15-4cf6-b5e7-e7129825d668\") " pod="openshift-multus/multus-2jc7k" Jul 13 22:13:27.292344 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:13:27.290750 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/01c7e40b-ef15-4cf6-b5e7-e7129825d668-cni-binary-copy\") pod \"multus-2jc7k\" (UID: \"01c7e40b-ef15-4cf6-b5e7-e7129825d668\") " pod="openshift-multus/multus-2jc7k" Jul 13 22:13:27.292344 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:13:27.290773 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/19a018fb-4dd4-4a76-9305-8f70a789d38d-systemd-units\") pod \"ovnkube-node-khllb\" (UID: \"19a018fb-4dd4-4a76-9305-8f70a789d38d\") " pod="openshift-ovn-kubernetes/ovnkube-node-khllb" Jul 13 22:13:27.292344 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:13:27.290813 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gtqs2\" (UniqueName: \"kubernetes.io/projected/5b796097-01b1-4a5d-a10e-2cf14087d82e-kube-api-access-gtqs2\") pod \"aws-ebs-csi-driver-node-2r94b\" (UID: \"5b796097-01b1-4a5d-a10e-2cf14087d82e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2r94b" Jul 13 22:13:27.292344 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:13:27.290839 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/01c7e40b-ef15-4cf6-b5e7-e7129825d668-multus-cni-dir\") pod \"multus-2jc7k\" (UID: \"01c7e40b-ef15-4cf6-b5e7-e7129825d668\") " pod="openshift-multus/multus-2jc7k" Jul 13 22:13:27.292344 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:13:27.290854 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/5b796097-01b1-4a5d-a10e-2cf14087d82e-kubelet-dir\") pod \"aws-ebs-csi-driver-node-2r94b\" (UID: \"5b796097-01b1-4a5d-a10e-2cf14087d82e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2r94b" Jul 13 22:13:27.292344 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:13:27.290871 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sxh8f\" (UniqueName: \"kubernetes.io/projected/e8e90a01-fd99-4a55-ba17-522966f4fabe-kube-api-access-sxh8f\") pod \"node-resolver-grjf8\" (UID: \"e8e90a01-fd99-4a55-ba17-522966f4fabe\") " pod="openshift-dns/node-resolver-grjf8" Jul 13 22:13:27.292344 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:13:27.290897 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/19a018fb-4dd4-4a76-9305-8f70a789d38d-etc-openvswitch\") pod \"ovnkube-node-khllb\" (UID: \"19a018fb-4dd4-4a76-9305-8f70a789d38d\") " pod="openshift-ovn-kubernetes/ovnkube-node-khllb" Jul 13 22:13:27.292344 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:13:27.290922 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/e05ae6ca-894a-4061-91ac-a0a6ed25d1b2-etc-systemd\") pod \"tuned-sht7p\" (UID: \"e05ae6ca-894a-4061-91ac-a0a6ed25d1b2\") " pod="openshift-cluster-node-tuning-operator/tuned-sht7p" Jul 13 22:13:27.292344 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:13:27.290935 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/03f58cdb-4d8b-41a9-aa82-dff50f158f0c-os-release\") pod \"multus-additional-cni-plugins-nbdsh\" (UID: \"03f58cdb-4d8b-41a9-aa82-dff50f158f0c\") " pod="openshift-multus/multus-additional-cni-plugins-nbdsh" Jul 13 22:13:27.292344 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:13:27.290964 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/e05ae6ca-894a-4061-91ac-a0a6ed25d1b2-lib-modules\") pod \"tuned-sht7p\" (UID: \"e05ae6ca-894a-4061-91ac-a0a6ed25d1b2\") " pod="openshift-cluster-node-tuning-operator/tuned-sht7p" Jul 13 22:13:27.292344 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:13:27.291445 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-45p8v\" (UniqueName: \"kubernetes.io/projected/54e8db97-3d72-4aa1-80bb-e56f10c7e474-kube-api-access-45p8v\") pod \"node-ca-r26w6\" (UID: \"54e8db97-3d72-4aa1-80bb-e56f10c7e474\") " pod="openshift-image-registry/node-ca-r26w6" Jul 13 22:13:27.292344 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:13:27.291483 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/5b796097-01b1-4a5d-a10e-2cf14087d82e-registration-dir\") pod \"aws-ebs-csi-driver-node-2r94b\" (UID: \"5b796097-01b1-4a5d-a10e-2cf14087d82e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2r94b" Jul 13 22:13:27.292344 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:13:27.291513 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/01c7e40b-ef15-4cf6-b5e7-e7129825d668-host-var-lib-cni-bin\") pod \"multus-2jc7k\" (UID: \"01c7e40b-ef15-4cf6-b5e7-e7129825d668\") " pod="openshift-multus/multus-2jc7k" Jul 13 22:13:27.292344 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:13:27.291541 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/19a018fb-4dd4-4a76-9305-8f70a789d38d-env-overrides\") pod \"ovnkube-node-khllb\" (UID: \"19a018fb-4dd4-4a76-9305-8f70a789d38d\") " pod="openshift-ovn-kubernetes/ovnkube-node-khllb" Jul 13 22:13:27.293503 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:13:27.291549 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/03f58cdb-4d8b-41a9-aa82-dff50f158f0c-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-nbdsh\" (UID: \"03f58cdb-4d8b-41a9-aa82-dff50f158f0c\") " pod="openshift-multus/multus-additional-cni-plugins-nbdsh" Jul 13 22:13:27.293503 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:13:27.291592 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/d38c850d-d65b-4f99-ad24-5c1d443bff4b-kubelet-config\") pod \"global-pull-secret-syncer-9hl5q\" (UID: \"d38c850d-d65b-4f99-ad24-5c1d443bff4b\") " pod="kube-system/global-pull-secret-syncer-9hl5q" Jul 13 22:13:27.293503 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:13:27.291633 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/19a018fb-4dd4-4a76-9305-8f70a789d38d-host-kubelet\") pod \"ovnkube-node-khllb\" (UID: \"19a018fb-4dd4-4a76-9305-8f70a789d38d\") " pod="openshift-ovn-kubernetes/ovnkube-node-khllb" Jul 13 22:13:27.293503 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:13:27.291675 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/19a018fb-4dd4-4a76-9305-8f70a789d38d-host-cni-bin\") pod \"ovnkube-node-khllb\" (UID: \"19a018fb-4dd4-4a76-9305-8f70a789d38d\") " pod="openshift-ovn-kubernetes/ovnkube-node-khllb" Jul 13 22:13:27.293503 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:13:27.291697 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/d38c850d-d65b-4f99-ad24-5c1d443bff4b-kubelet-config\") pod \"global-pull-secret-syncer-9hl5q\" (UID: \"d38c850d-d65b-4f99-ad24-5c1d443bff4b\") " pod="kube-system/global-pull-secret-syncer-9hl5q" Jul 13 22:13:27.293503 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:13:27.291716 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/19a018fb-4dd4-4a76-9305-8f70a789d38d-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-khllb\" (UID: \"19a018fb-4dd4-4a76-9305-8f70a789d38d\") " pod="openshift-ovn-kubernetes/ovnkube-node-khllb" Jul 13 22:13:27.293503 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:13:27.291741 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/3357149c-54b6-4e57-93f0-2b9bf43ea301-host-slash\") pod \"iptables-alerter-brfkn\" (UID: \"3357149c-54b6-4e57-93f0-2b9bf43ea301\") " pod="openshift-network-operator/iptables-alerter-brfkn" Jul 13 22:13:27.293503 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:13:27.291789 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/e05ae6ca-894a-4061-91ac-a0a6ed25d1b2-host\") pod \"tuned-sht7p\" (UID: \"e05ae6ca-894a-4061-91ac-a0a6ed25d1b2\") " pod="openshift-cluster-node-tuning-operator/tuned-sht7p" Jul 13 22:13:27.293503 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:13:27.292014 2583 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory" Jul 13 22:13:27.293503 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:13:27.292248 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/ad16aa2f-aede-4d47-82d6-0ab7dda663d4-konnectivity-ca\") pod \"konnectivity-agent-wgcj4\" (UID: \"ad16aa2f-aede-4d47-82d6-0ab7dda663d4\") " pod="kube-system/konnectivity-agent-wgcj4" Jul 13 22:13:27.293503 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:13:27.292311 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/19a018fb-4dd4-4a76-9305-8f70a789d38d-host-run-netns\") pod \"ovnkube-node-khllb\" (UID: \"19a018fb-4dd4-4a76-9305-8f70a789d38d\") " pod="openshift-ovn-kubernetes/ovnkube-node-khllb" Jul 13 22:13:27.293503 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:13:27.292330 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/01c7e40b-ef15-4cf6-b5e7-e7129825d668-etc-kubernetes\") pod \"multus-2jc7k\" (UID: \"01c7e40b-ef15-4cf6-b5e7-e7129825d668\") " pod="openshift-multus/multus-2jc7k" Jul 13 22:13:27.293503 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:13:27.292354 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/19a018fb-4dd4-4a76-9305-8f70a789d38d-log-socket\") pod \"ovnkube-node-khllb\" (UID: \"19a018fb-4dd4-4a76-9305-8f70a789d38d\") " pod="openshift-ovn-kubernetes/ovnkube-node-khllb" Jul 13 22:13:27.293503 ip-10-0-133-179 kubenswrapper[2583]: E0713 22:13:27.292417 2583 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 13 22:13:27.293503 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:13:27.292452 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/e05ae6ca-894a-4061-91ac-a0a6ed25d1b2-etc-modprobe-d\") pod \"tuned-sht7p\" (UID: \"e05ae6ca-894a-4061-91ac-a0a6ed25d1b2\") " pod="openshift-cluster-node-tuning-operator/tuned-sht7p" Jul 13 22:13:27.293503 ip-10-0-133-179 kubenswrapper[2583]: E0713 22:13:27.292500 2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d38c850d-d65b-4f99-ad24-5c1d443bff4b-original-pull-secret podName:d38c850d-d65b-4f99-ad24-5c1d443bff4b nodeName:}" failed. No retries permitted until 2026-07-13 22:13:27.792479232 +0000 UTC m=+3.040394581 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/d38c850d-d65b-4f99-ad24-5c1d443bff4b-original-pull-secret") pod "global-pull-secret-syncer-9hl5q" (UID: "d38c850d-d65b-4f99-ad24-5c1d443bff4b") : object "kube-system"/"original-pull-secret" not registered Jul 13 22:13:27.293503 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:13:27.292501 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/e05ae6ca-894a-4061-91ac-a0a6ed25d1b2-etc-kubernetes\") pod \"tuned-sht7p\" (UID: \"e05ae6ca-894a-4061-91ac-a0a6ed25d1b2\") " pod="openshift-cluster-node-tuning-operator/tuned-sht7p" Jul 13 22:13:27.294391 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:13:27.292545 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/5b796097-01b1-4a5d-a10e-2cf14087d82e-device-dir\") pod \"aws-ebs-csi-driver-node-2r94b\" (UID: \"5b796097-01b1-4a5d-a10e-2cf14087d82e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2r94b" Jul 13 22:13:27.294391 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:13:27.292695 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/01c7e40b-ef15-4cf6-b5e7-e7129825d668-system-cni-dir\") pod \"multus-2jc7k\" (UID: \"01c7e40b-ef15-4cf6-b5e7-e7129825d668\") " pod="openshift-multus/multus-2jc7k" Jul 13 22:13:27.294391 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:13:27.292737 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/d38c850d-d65b-4f99-ad24-5c1d443bff4b-dbus\") pod \"global-pull-secret-syncer-9hl5q\" (UID: \"d38c850d-d65b-4f99-ad24-5c1d443bff4b\") " pod="kube-system/global-pull-secret-syncer-9hl5q" Jul 13 22:13:27.294391 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:13:27.292741 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/01c7e40b-ef15-4cf6-b5e7-e7129825d668-host-var-lib-cni-multus\") pod \"multus-2jc7k\" (UID: \"01c7e40b-ef15-4cf6-b5e7-e7129825d668\") " pod="openshift-multus/multus-2jc7k" Jul 13 22:13:27.294391 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:13:27.292832 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/5b796097-01b1-4a5d-a10e-2cf14087d82e-socket-dir\") pod \"aws-ebs-csi-driver-node-2r94b\" (UID: \"5b796097-01b1-4a5d-a10e-2cf14087d82e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2r94b" Jul 13 22:13:27.294391 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:13:27.292874 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/03f58cdb-4d8b-41a9-aa82-dff50f158f0c-system-cni-dir\") pod \"multus-additional-cni-plugins-nbdsh\" (UID: \"03f58cdb-4d8b-41a9-aa82-dff50f158f0c\") " pod="openshift-multus/multus-additional-cni-plugins-nbdsh" Jul 13 22:13:27.294391 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:13:27.293110 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e8e90a01-fd99-4a55-ba17-522966f4fabe-tmp-dir\") pod \"node-resolver-grjf8\" (UID: \"e8e90a01-fd99-4a55-ba17-522966f4fabe\") " pod="openshift-dns/node-resolver-grjf8" Jul 13 22:13:27.294391 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:13:27.293168 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/03f58cdb-4d8b-41a9-aa82-dff50f158f0c-cni-binary-copy\") pod \"multus-additional-cni-plugins-nbdsh\" (UID: \"03f58cdb-4d8b-41a9-aa82-dff50f158f0c\") " pod="openshift-multus/multus-additional-cni-plugins-nbdsh" Jul 13 22:13:27.294391 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:13:27.293183 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/19a018fb-4dd4-4a76-9305-8f70a789d38d-host-slash\") pod \"ovnkube-node-khllb\" (UID: \"19a018fb-4dd4-4a76-9305-8f70a789d38d\") " pod="openshift-ovn-kubernetes/ovnkube-node-khllb" Jul 13 22:13:27.294391 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:13:27.293230 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/e05ae6ca-894a-4061-91ac-a0a6ed25d1b2-var-lib-kubelet\") pod \"tuned-sht7p\" (UID: \"e05ae6ca-894a-4061-91ac-a0a6ed25d1b2\") " pod="openshift-cluster-node-tuning-operator/tuned-sht7p" Jul 13 22:13:27.294391 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:13:27.293269 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/19a018fb-4dd4-4a76-9305-8f70a789d38d-run-openvswitch\") pod \"ovnkube-node-khllb\" (UID: \"19a018fb-4dd4-4a76-9305-8f70a789d38d\") " pod="openshift-ovn-kubernetes/ovnkube-node-khllb" Jul 13 22:13:27.294391 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:13:27.293277 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/19a018fb-4dd4-4a76-9305-8f70a789d38d-host-run-ovn-kubernetes\") pod \"ovnkube-node-khllb\" (UID: \"19a018fb-4dd4-4a76-9305-8f70a789d38d\") " pod="openshift-ovn-kubernetes/ovnkube-node-khllb" Jul 13 22:13:27.294391 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:13:27.293321 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/03f58cdb-4d8b-41a9-aa82-dff50f158f0c-cnibin\") pod \"multus-additional-cni-plugins-nbdsh\" (UID: \"03f58cdb-4d8b-41a9-aa82-dff50f158f0c\") " pod="openshift-multus/multus-additional-cni-plugins-nbdsh" Jul 13 22:13:27.294391 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:13:27.293362 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/5b796097-01b1-4a5d-a10e-2cf14087d82e-sys-fs\") pod \"aws-ebs-csi-driver-node-2r94b\" (UID: \"5b796097-01b1-4a5d-a10e-2cf14087d82e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2r94b" Jul 13 22:13:27.294391 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:13:27.293366 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/01c7e40b-ef15-4cf6-b5e7-e7129825d668-multus-socket-dir-parent\") pod \"multus-2jc7k\" (UID: \"01c7e40b-ef15-4cf6-b5e7-e7129825d668\") " pod="openshift-multus/multus-2jc7k" Jul 13 22:13:27.294391 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:13:27.293429 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/01c7e40b-ef15-4cf6-b5e7-e7129825d668-cnibin\") pod \"multus-2jc7k\" (UID: \"01c7e40b-ef15-4cf6-b5e7-e7129825d668\") " pod="openshift-multus/multus-2jc7k" Jul 13 22:13:27.294391 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:13:27.293469 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/19a018fb-4dd4-4a76-9305-8f70a789d38d-run-ovn\") pod \"ovnkube-node-khllb\" (UID: \"19a018fb-4dd4-4a76-9305-8f70a789d38d\") " pod="openshift-ovn-kubernetes/ovnkube-node-khllb" Jul 13 22:13:27.302503 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:13:27.297090 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/e05ae6ca-894a-4061-91ac-a0a6ed25d1b2-run\") pod \"tuned-sht7p\" (UID: \"e05ae6ca-894a-4061-91ac-a0a6ed25d1b2\") " pod="openshift-cluster-node-tuning-operator/tuned-sht7p" Jul 13 22:13:27.302503 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:13:27.297167 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/54e8db97-3d72-4aa1-80bb-e56f10c7e474-host\") pod \"node-ca-r26w6\" (UID: \"54e8db97-3d72-4aa1-80bb-e56f10c7e474\") " pod="openshift-image-registry/node-ca-r26w6" Jul 13 22:13:27.302503 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:13:27.297619 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/54e8db97-3d72-4aa1-80bb-e56f10c7e474-serviceca\") pod \"node-ca-r26w6\" (UID: \"54e8db97-3d72-4aa1-80bb-e56f10c7e474\") " pod="openshift-image-registry/node-ca-r26w6" Jul 13 22:13:27.302503 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:13:27.297686 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/01c7e40b-ef15-4cf6-b5e7-e7129825d668-host-run-netns\") pod \"multus-2jc7k\" (UID: \"01c7e40b-ef15-4cf6-b5e7-e7129825d668\") " pod="openshift-multus/multus-2jc7k" Jul 13 22:13:27.302503 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:13:27.301495 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/e05ae6ca-894a-4061-91ac-a0a6ed25d1b2-etc-tuned\") pod \"tuned-sht7p\" (UID: \"e05ae6ca-894a-4061-91ac-a0a6ed25d1b2\") " pod="openshift-cluster-node-tuning-operator/tuned-sht7p" Jul 13 22:13:27.302503 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:13:27.302300 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/5b796097-01b1-4a5d-a10e-2cf14087d82e-etc-selinux\") pod \"aws-ebs-csi-driver-node-2r94b\" (UID: \"5b796097-01b1-4a5d-a10e-2cf14087d82e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2r94b" Jul 13 22:13:27.302503 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:13:27.302378 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/01c7e40b-ef15-4cf6-b5e7-e7129825d668-host-var-lib-kubelet\") pod \"multus-2jc7k\" (UID: \"01c7e40b-ef15-4cf6-b5e7-e7129825d668\") " pod="openshift-multus/multus-2jc7k" Jul 13 22:13:27.302980 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:13:27.302761 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/e05ae6ca-894a-4061-91ac-a0a6ed25d1b2-sys\") pod \"tuned-sht7p\" (UID: \"e05ae6ca-894a-4061-91ac-a0a6ed25d1b2\") " pod="openshift-cluster-node-tuning-operator/tuned-sht7p" Jul 13 22:13:27.302980 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:13:27.302766 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/19a018fb-4dd4-4a76-9305-8f70a789d38d-systemd-units\") pod \"ovnkube-node-khllb\" (UID: \"19a018fb-4dd4-4a76-9305-8f70a789d38d\") " pod="openshift-ovn-kubernetes/ovnkube-node-khllb" Jul 13 22:13:27.302980 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:13:27.302966 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/19a018fb-4dd4-4a76-9305-8f70a789d38d-run-systemd\") pod \"ovnkube-node-khllb\" (UID: \"19a018fb-4dd4-4a76-9305-8f70a789d38d\") " pod="openshift-ovn-kubernetes/ovnkube-node-khllb" Jul 13 22:13:27.303120 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:13:27.302970 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/01c7e40b-ef15-4cf6-b5e7-e7129825d668-multus-cni-dir\") pod \"multus-2jc7k\" (UID: \"01c7e40b-ef15-4cf6-b5e7-e7129825d668\") " pod="openshift-multus/multus-2jc7k" Jul 13 22:13:27.303120 ip-10-0-133-179 kubenswrapper[2583]: E0713 22:13:27.303094 2583 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 13 22:13:27.303212 ip-10-0-133-179 kubenswrapper[2583]: E0713 22:13:27.303157 2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/85265522-e114-4c04-b595-a26fcae86884-metrics-certs podName:85265522-e114-4c04-b595-a26fcae86884 nodeName:}" failed. No retries permitted until 2026-07-13 22:13:27.803137993 +0000 UTC m=+3.051053336 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/85265522-e114-4c04-b595-a26fcae86884-metrics-certs") pod "network-metrics-daemon-26w5h" (UID: "85265522-e114-4c04-b595-a26fcae86884") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 13 22:13:27.303303 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:13:27.303279 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/19a018fb-4dd4-4a76-9305-8f70a789d38d-var-lib-openvswitch\") pod \"ovnkube-node-khllb\" (UID: \"19a018fb-4dd4-4a76-9305-8f70a789d38d\") " pod="openshift-ovn-kubernetes/ovnkube-node-khllb" Jul 13 22:13:27.303356 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:13:27.303338 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/19a018fb-4dd4-4a76-9305-8f70a789d38d-host-cni-netd\") pod \"ovnkube-node-khllb\" (UID: \"19a018fb-4dd4-4a76-9305-8f70a789d38d\") " pod="openshift-ovn-kubernetes/ovnkube-node-khllb" Jul 13 22:13:27.303968 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:13:27.303645 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/01c7e40b-ef15-4cf6-b5e7-e7129825d668-multus-daemon-config\") pod \"multus-2jc7k\" (UID: \"01c7e40b-ef15-4cf6-b5e7-e7129825d668\") " pod="openshift-multus/multus-2jc7k" Jul 13 22:13:27.303968 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:13:27.303714 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/19a018fb-4dd4-4a76-9305-8f70a789d38d-etc-openvswitch\") pod \"ovnkube-node-khllb\" (UID: \"19a018fb-4dd4-4a76-9305-8f70a789d38d\") " pod="openshift-ovn-kubernetes/ovnkube-node-khllb" Jul 13 22:13:27.303968 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:13:27.303710 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/5b796097-01b1-4a5d-a10e-2cf14087d82e-registration-dir\") pod \"aws-ebs-csi-driver-node-2r94b\" (UID: \"5b796097-01b1-4a5d-a10e-2cf14087d82e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2r94b" Jul 13 22:13:27.303968 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:13:27.303769 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/e05ae6ca-894a-4061-91ac-a0a6ed25d1b2-etc-systemd\") pod \"tuned-sht7p\" (UID: \"e05ae6ca-894a-4061-91ac-a0a6ed25d1b2\") " pod="openshift-cluster-node-tuning-operator/tuned-sht7p" Jul 13 22:13:27.303968 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:13:27.303796 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/01c7e40b-ef15-4cf6-b5e7-e7129825d668-host-var-lib-cni-bin\") pod \"multus-2jc7k\" (UID: \"01c7e40b-ef15-4cf6-b5e7-e7129825d668\") " pod="openshift-multus/multus-2jc7k" Jul 13 22:13:27.303968 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:13:27.303925 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/01c7e40b-ef15-4cf6-b5e7-e7129825d668-cni-binary-copy\") pod \"multus-2jc7k\" (UID: \"01c7e40b-ef15-4cf6-b5e7-e7129825d668\") " pod="openshift-multus/multus-2jc7k" Jul 13 22:13:27.304367 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:13:27.304032 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/19a018fb-4dd4-4a76-9305-8f70a789d38d-node-log\") pod \"ovnkube-node-khllb\" (UID: \"19a018fb-4dd4-4a76-9305-8f70a789d38d\") " pod="openshift-ovn-kubernetes/ovnkube-node-khllb" Jul 13 22:13:27.304367 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:13:27.293230 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/e8e90a01-fd99-4a55-ba17-522966f4fabe-hosts-file\") pod \"node-resolver-grjf8\" (UID: \"e8e90a01-fd99-4a55-ba17-522966f4fabe\") " pod="openshift-dns/node-resolver-grjf8" Jul 13 22:13:27.304464 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:13:27.304383 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/19a018fb-4dd4-4a76-9305-8f70a789d38d-env-overrides\") pod \"ovnkube-node-khllb\" (UID: \"19a018fb-4dd4-4a76-9305-8f70a789d38d\") " pod="openshift-ovn-kubernetes/ovnkube-node-khllb" Jul 13 22:13:27.304464 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:13:27.293320 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/01c7e40b-ef15-4cf6-b5e7-e7129825d668-host-run-multus-certs\") pod \"multus-2jc7k\" (UID: \"01c7e40b-ef15-4cf6-b5e7-e7129825d668\") " pod="openshift-multus/multus-2jc7k" Jul 13 22:13:27.304589 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:13:27.304497 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/01c7e40b-ef15-4cf6-b5e7-e7129825d668-host-run-k8s-cni-cncf-io\") pod \"multus-2jc7k\" (UID: \"01c7e40b-ef15-4cf6-b5e7-e7129825d668\") " pod="openshift-multus/multus-2jc7k" Jul 13 22:13:27.304648 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:13:27.304633 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/03f58cdb-4d8b-41a9-aa82-dff50f158f0c-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-nbdsh\" (UID: \"03f58cdb-4d8b-41a9-aa82-dff50f158f0c\") " pod="openshift-multus/multus-additional-cni-plugins-nbdsh" Jul 13 22:13:27.305094 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:13:27.305062 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/19a018fb-4dd4-4a76-9305-8f70a789d38d-ovnkube-config\") pod \"ovnkube-node-khllb\" (UID: \"19a018fb-4dd4-4a76-9305-8f70a789d38d\") " pod="openshift-ovn-kubernetes/ovnkube-node-khllb" Jul 13 22:13:27.305193 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:13:27.305163 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/e05ae6ca-894a-4061-91ac-a0a6ed25d1b2-etc-sysconfig\") pod \"tuned-sht7p\" (UID: \"e05ae6ca-894a-4061-91ac-a0a6ed25d1b2\") " pod="openshift-cluster-node-tuning-operator/tuned-sht7p" Jul 13 22:13:27.305326 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:13:27.305309 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/e05ae6ca-894a-4061-91ac-a0a6ed25d1b2-etc-sysctl-conf\") pod \"tuned-sht7p\" (UID: \"e05ae6ca-894a-4061-91ac-a0a6ed25d1b2\") " pod="openshift-cluster-node-tuning-operator/tuned-sht7p" Jul 13 22:13:27.305473 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:13:27.305447 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/e05ae6ca-894a-4061-91ac-a0a6ed25d1b2-lib-modules\") pod \"tuned-sht7p\" (UID: \"e05ae6ca-894a-4061-91ac-a0a6ed25d1b2\") " pod="openshift-cluster-node-tuning-operator/tuned-sht7p" Jul 13 22:13:27.305638 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:13:27.305584 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/e05ae6ca-894a-4061-91ac-a0a6ed25d1b2-etc-sysctl-d\") pod \"tuned-sht7p\" (UID: \"e05ae6ca-894a-4061-91ac-a0a6ed25d1b2\") " pod="openshift-cluster-node-tuning-operator/tuned-sht7p" Jul 13 22:13:27.307178 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:13:27.307155 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/3357149c-54b6-4e57-93f0-2b9bf43ea301-iptables-alerter-script\") pod \"iptables-alerter-brfkn\" (UID: \"3357149c-54b6-4e57-93f0-2b9bf43ea301\") " pod="openshift-network-operator/iptables-alerter-brfkn" Jul 13 22:13:27.307283 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:13:27.307255 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zwfk5\" (UniqueName: \"kubernetes.io/projected/85265522-e114-4c04-b595-a26fcae86884-kube-api-access-zwfk5\") pod \"network-metrics-daemon-26w5h\" (UID: \"85265522-e114-4c04-b595-a26fcae86884\") " pod="openshift-multus/network-metrics-daemon-26w5h" Jul 13 22:13:27.307452 ip-10-0-133-179 kubenswrapper[2583]: E0713 22:13:27.307427 2583 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 13 22:13:27.307522 ip-10-0-133-179 kubenswrapper[2583]: E0713 22:13:27.307458 2583 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 13 22:13:27.307522 ip-10-0-133-179 kubenswrapper[2583]: E0713 22:13:27.307474 2583 projected.go:196] Error preparing data for projected volume kube-api-access-tcrhd for pod openshift-network-diagnostics/network-check-target-v8mkx: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 13 22:13:27.307662 ip-10-0-133-179 kubenswrapper[2583]: E0713 22:13:27.307544 2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/c8bbc3a9-6b5c-485d-b7ef-ca4cdca462ce-kube-api-access-tcrhd podName:c8bbc3a9-6b5c-485d-b7ef-ca4cdca462ce nodeName:}" failed. No retries permitted until 2026-07-13 22:13:27.807526276 +0000 UTC m=+3.055441623 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-tcrhd" (UniqueName: "kubernetes.io/projected/c8bbc3a9-6b5c-485d-b7ef-ca4cdca462ce-kube-api-access-tcrhd") pod "network-check-target-v8mkx" (UID: "c8bbc3a9-6b5c-485d-b7ef-ca4cdca462ce") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 13 22:13:27.310355 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:13:27.310329 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/19a018fb-4dd4-4a76-9305-8f70a789d38d-ovn-node-metrics-cert\") pod \"ovnkube-node-khllb\" (UID: \"19a018fb-4dd4-4a76-9305-8f70a789d38d\") " pod="openshift-ovn-kubernetes/ovnkube-node-khllb" Jul 13 22:13:27.310532 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:13:27.310514 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/e05ae6ca-894a-4061-91ac-a0a6ed25d1b2-tmp\") pod \"tuned-sht7p\" (UID: \"e05ae6ca-894a-4061-91ac-a0a6ed25d1b2\") " pod="openshift-cluster-node-tuning-operator/tuned-sht7p" Jul 13 22:13:27.310616 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:13:27.310517 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6fxhd\" (UniqueName: \"kubernetes.io/projected/03f58cdb-4d8b-41a9-aa82-dff50f158f0c-kube-api-access-6fxhd\") pod \"multus-additional-cni-plugins-nbdsh\" (UID: \"03f58cdb-4d8b-41a9-aa82-dff50f158f0c\") " pod="openshift-multus/multus-additional-cni-plugins-nbdsh" Jul 13 22:13:27.311498 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:13:27.311471 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/ad16aa2f-aede-4d47-82d6-0ab7dda663d4-agent-certs\") pod \"konnectivity-agent-wgcj4\" (UID: \"ad16aa2f-aede-4d47-82d6-0ab7dda663d4\") " pod="kube-system/konnectivity-agent-wgcj4" Jul 13 22:13:27.312511 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:13:27.312489 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gtqs2\" (UniqueName: \"kubernetes.io/projected/5b796097-01b1-4a5d-a10e-2cf14087d82e-kube-api-access-gtqs2\") pod \"aws-ebs-csi-driver-node-2r94b\" (UID: \"5b796097-01b1-4a5d-a10e-2cf14087d82e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2r94b" Jul 13 22:13:27.314974 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:13:27.314949 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xkrxz\" (UniqueName: \"kubernetes.io/projected/01c7e40b-ef15-4cf6-b5e7-e7129825d668-kube-api-access-xkrxz\") pod \"multus-2jc7k\" (UID: \"01c7e40b-ef15-4cf6-b5e7-e7129825d668\") " pod="openshift-multus/multus-2jc7k" Jul 13 22:13:27.315112 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:13:27.315093 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-w5f8s\" (UniqueName: \"kubernetes.io/projected/e05ae6ca-894a-4061-91ac-a0a6ed25d1b2-kube-api-access-w5f8s\") pod \"tuned-sht7p\" (UID: \"e05ae6ca-894a-4061-91ac-a0a6ed25d1b2\") " pod="openshift-cluster-node-tuning-operator/tuned-sht7p" Jul 13 22:13:27.319484 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:13:27.319434 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xf67g\" (UniqueName: \"kubernetes.io/projected/3357149c-54b6-4e57-93f0-2b9bf43ea301-kube-api-access-xf67g\") pod \"iptables-alerter-brfkn\" (UID: \"3357149c-54b6-4e57-93f0-2b9bf43ea301\") " pod="openshift-network-operator/iptables-alerter-brfkn" Jul 13 22:13:27.320090 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:13:27.320065 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sxh8f\" (UniqueName: \"kubernetes.io/projected/e8e90a01-fd99-4a55-ba17-522966f4fabe-kube-api-access-sxh8f\") pod \"node-resolver-grjf8\" (UID: \"e8e90a01-fd99-4a55-ba17-522966f4fabe\") " pod="openshift-dns/node-resolver-grjf8" Jul 13 22:13:27.320888 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:13:27.320858 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-45p8v\" (UniqueName: \"kubernetes.io/projected/54e8db97-3d72-4aa1-80bb-e56f10c7e474-kube-api-access-45p8v\") pod \"node-ca-r26w6\" (UID: \"54e8db97-3d72-4aa1-80bb-e56f10c7e474\") " pod="openshift-image-registry/node-ca-r26w6" Jul 13 22:13:27.321481 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:13:27.321459 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-47txq\" (UniqueName: \"kubernetes.io/projected/19a018fb-4dd4-4a76-9305-8f70a789d38d-kube-api-access-47txq\") pod \"ovnkube-node-khllb\" (UID: \"19a018fb-4dd4-4a76-9305-8f70a789d38d\") " pod="openshift-ovn-kubernetes/ovnkube-node-khllb" Jul 13 22:13:27.391533 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:13:27.391475 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-179.ec2.internal" event={"ID":"8bd03286da06035aeb817bb940fc3daa","Type":"ContainerStarted","Data":"e7a6588c4036cd616befd7cf65ee0b76f8cdd0533bbe11804c648fd2b6350201"} Jul 13 22:13:27.393452 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:13:27.393423 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-133-179.ec2.internal" event={"ID":"120f6b7b6960a904bf802202dca0166e","Type":"ContainerStarted","Data":"231630cc5d3e097bd5095f34a342ca662a4bb12a3e0ee606a320bfe89bdfd86d"} Jul 13 22:13:27.471173 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:13:27.470857 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-grjf8" Jul 13 22:13:27.479940 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:13:27.479852 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-r26w6" Jul 13 22:13:27.491163 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:13:27.491132 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-brfkn" Jul 13 22:13:27.496456 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:13:27.496426 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-khllb" Jul 13 22:13:27.502733 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:13:27.502541 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-sht7p" Jul 13 22:13:27.510891 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:13:27.510867 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2r94b" Jul 13 22:13:27.517931 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:13:27.517906 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-wgcj4" Jul 13 22:13:27.526040 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:13:27.525925 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-2jc7k" Jul 13 22:13:27.532692 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:13:27.532661 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-nbdsh" Jul 13 22:13:27.794468 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:13:27.794400 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/d38c850d-d65b-4f99-ad24-5c1d443bff4b-original-pull-secret\") pod \"global-pull-secret-syncer-9hl5q\" (UID: \"d38c850d-d65b-4f99-ad24-5c1d443bff4b\") " pod="kube-system/global-pull-secret-syncer-9hl5q" Jul 13 22:13:27.794639 ip-10-0-133-179 kubenswrapper[2583]: E0713 22:13:27.794591 2583 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 13 22:13:27.794704 ip-10-0-133-179 kubenswrapper[2583]: E0713 22:13:27.794672 2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d38c850d-d65b-4f99-ad24-5c1d443bff4b-original-pull-secret podName:d38c850d-d65b-4f99-ad24-5c1d443bff4b nodeName:}" failed. No retries permitted until 2026-07-13 22:13:28.794651075 +0000 UTC m=+4.042566417 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/d38c850d-d65b-4f99-ad24-5c1d443bff4b-original-pull-secret") pod "global-pull-secret-syncer-9hl5q" (UID: "d38c850d-d65b-4f99-ad24-5c1d443bff4b") : object "kube-system"/"original-pull-secret" not registered Jul 13 22:13:27.895202 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:13:27.895160 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tcrhd\" (UniqueName: \"kubernetes.io/projected/c8bbc3a9-6b5c-485d-b7ef-ca4cdca462ce-kube-api-access-tcrhd\") pod \"network-check-target-v8mkx\" (UID: \"c8bbc3a9-6b5c-485d-b7ef-ca4cdca462ce\") " pod="openshift-network-diagnostics/network-check-target-v8mkx" Jul 13 22:13:27.895202 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:13:27.895208 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/85265522-e114-4c04-b595-a26fcae86884-metrics-certs\") pod \"network-metrics-daemon-26w5h\" (UID: \"85265522-e114-4c04-b595-a26fcae86884\") " pod="openshift-multus/network-metrics-daemon-26w5h" Jul 13 22:13:27.895449 ip-10-0-133-179 kubenswrapper[2583]: E0713 22:13:27.895342 2583 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 13 22:13:27.895449 ip-10-0-133-179 kubenswrapper[2583]: E0713 22:13:27.895356 2583 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 13 22:13:27.895449 ip-10-0-133-179 kubenswrapper[2583]: E0713 22:13:27.895380 2583 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 13 22:13:27.895449 ip-10-0-133-179 kubenswrapper[2583]: E0713 22:13:27.895390 2583 projected.go:196] Error preparing data for projected volume kube-api-access-tcrhd for pod openshift-network-diagnostics/network-check-target-v8mkx: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 13 22:13:27.895449 ip-10-0-133-179 kubenswrapper[2583]: E0713 22:13:27.895409 2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/85265522-e114-4c04-b595-a26fcae86884-metrics-certs podName:85265522-e114-4c04-b595-a26fcae86884 nodeName:}" failed. No retries permitted until 2026-07-13 22:13:28.895389679 +0000 UTC m=+4.143305008 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/85265522-e114-4c04-b595-a26fcae86884-metrics-certs") pod "network-metrics-daemon-26w5h" (UID: "85265522-e114-4c04-b595-a26fcae86884") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 13 22:13:27.895650 ip-10-0-133-179 kubenswrapper[2583]: E0713 22:13:27.895456 2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/c8bbc3a9-6b5c-485d-b7ef-ca4cdca462ce-kube-api-access-tcrhd podName:c8bbc3a9-6b5c-485d-b7ef-ca4cdca462ce nodeName:}" failed. No retries permitted until 2026-07-13 22:13:28.8954245 +0000 UTC m=+4.143339828 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-tcrhd" (UniqueName: "kubernetes.io/projected/c8bbc3a9-6b5c-485d-b7ef-ca4cdca462ce-kube-api-access-tcrhd") pod "network-check-target-v8mkx" (UID: "c8bbc3a9-6b5c-485d-b7ef-ca4cdca462ce") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 13 22:13:28.225773 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:13:28.225694 2583 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-12 22:08:27 +0000 UTC" deadline="2028-01-29 19:36:45.289617896 +0000 UTC" Jul 13 22:13:28.225773 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:13:28.225725 2583 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13557h23m17.063895581s" Jul 13 22:13:28.434466 ip-10-0-133-179 kubenswrapper[2583]: W0713 22:13:28.434434 2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod03f58cdb_4d8b_41a9_aa82_dff50f158f0c.slice/crio-8ecc0d34a0dd6f0fd5b1a2a389a9e3fd7d507edd91c2035deb2ab4bdb1b1e3a5 WatchSource:0}: Error finding container 8ecc0d34a0dd6f0fd5b1a2a389a9e3fd7d507edd91c2035deb2ab4bdb1b1e3a5: Status 404 returned error can't find the container with id 8ecc0d34a0dd6f0fd5b1a2a389a9e3fd7d507edd91c2035deb2ab4bdb1b1e3a5 Jul 13 22:13:28.437684 ip-10-0-133-179 kubenswrapper[2583]: W0713 22:13:28.436940 2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod54e8db97_3d72_4aa1_80bb_e56f10c7e474.slice/crio-29c71c08f25dc0727f0fa67246158400b06ded3a10394e7c2b9ed06b83ee1f3a WatchSource:0}: Error finding container 29c71c08f25dc0727f0fa67246158400b06ded3a10394e7c2b9ed06b83ee1f3a: Status 404 returned error can't find the container with id 29c71c08f25dc0727f0fa67246158400b06ded3a10394e7c2b9ed06b83ee1f3a Jul 13 22:13:28.441348 ip-10-0-133-179 kubenswrapper[2583]: W0713 22:13:28.441327 2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod19a018fb_4dd4_4a76_9305_8f70a789d38d.slice/crio-0dcfa99e8fc919f7af6f1a4205eaa2faa01f25c6ba847862d5a059a1aeabcdec WatchSource:0}: Error finding container 0dcfa99e8fc919f7af6f1a4205eaa2faa01f25c6ba847862d5a059a1aeabcdec: Status 404 returned error can't find the container with id 0dcfa99e8fc919f7af6f1a4205eaa2faa01f25c6ba847862d5a059a1aeabcdec Jul 13 22:13:28.461563 ip-10-0-133-179 kubenswrapper[2583]: W0713 22:13:28.461516 2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3357149c_54b6_4e57_93f0_2b9bf43ea301.slice/crio-b50df6302a11e5aa085e12c743a3f057066fdd2a6151b38ddd09eef5b41c6008 WatchSource:0}: Error finding container b50df6302a11e5aa085e12c743a3f057066fdd2a6151b38ddd09eef5b41c6008: Status 404 returned error can't find the container with id b50df6302a11e5aa085e12c743a3f057066fdd2a6151b38ddd09eef5b41c6008 Jul 13 22:13:28.462281 ip-10-0-133-179 kubenswrapper[2583]: W0713 22:13:28.462261 2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode8e90a01_fd99_4a55_ba17_522966f4fabe.slice/crio-ed612688796d264dfc5c38af4870ae086f611d2e173557adb108846ab89f35d3 WatchSource:0}: Error finding container ed612688796d264dfc5c38af4870ae086f611d2e173557adb108846ab89f35d3: Status 404 returned error can't find the container with id ed612688796d264dfc5c38af4870ae086f611d2e173557adb108846ab89f35d3 Jul 13 22:13:28.463124 ip-10-0-133-179 kubenswrapper[2583]: W0713 22:13:28.463041 2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode05ae6ca_894a_4061_91ac_a0a6ed25d1b2.slice/crio-7e6c1d9d68ded658d0ef8e8334db1d1d13248acd5a7f14ab51eac984b4ed3d9c WatchSource:0}: Error finding container 7e6c1d9d68ded658d0ef8e8334db1d1d13248acd5a7f14ab51eac984b4ed3d9c: Status 404 returned error can't find the container with id 7e6c1d9d68ded658d0ef8e8334db1d1d13248acd5a7f14ab51eac984b4ed3d9c Jul 13 22:13:28.464705 ip-10-0-133-179 kubenswrapper[2583]: W0713 22:13:28.464662 2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5b796097_01b1_4a5d_a10e_2cf14087d82e.slice/crio-1d7f5485cc6255a2fcbdba27b611d5765dfb03b396d360461d9cb7faec2c3e5a WatchSource:0}: Error finding container 1d7f5485cc6255a2fcbdba27b611d5765dfb03b396d360461d9cb7faec2c3e5a: Status 404 returned error can't find the container with id 1d7f5485cc6255a2fcbdba27b611d5765dfb03b396d360461d9cb7faec2c3e5a Jul 13 22:13:28.466043 ip-10-0-133-179 kubenswrapper[2583]: W0713 22:13:28.466010 2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podad16aa2f_aede_4d47_82d6_0ab7dda663d4.slice/crio-6e5700b5f2386246cd9e7689a6ec31795d9212ef16edf94f6d3209890f709561 WatchSource:0}: Error finding container 6e5700b5f2386246cd9e7689a6ec31795d9212ef16edf94f6d3209890f709561: Status 404 returned error can't find the container with id 6e5700b5f2386246cd9e7689a6ec31795d9212ef16edf94f6d3209890f709561 Jul 13 22:13:28.466674 ip-10-0-133-179 kubenswrapper[2583]: W0713 22:13:28.466650 2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod01c7e40b_ef15_4cf6_b5e7_e7129825d668.slice/crio-e9ba48e5b68010bbe988d2683939d9331d02701238fa16af3cd4f4849c165f3c WatchSource:0}: Error finding container e9ba48e5b68010bbe988d2683939d9331d02701238fa16af3cd4f4849c165f3c: Status 404 returned error can't find the container with id e9ba48e5b68010bbe988d2683939d9331d02701238fa16af3cd4f4849c165f3c Jul 13 22:13:28.802906 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:13:28.802658 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/d38c850d-d65b-4f99-ad24-5c1d443bff4b-original-pull-secret\") pod \"global-pull-secret-syncer-9hl5q\" (UID: \"d38c850d-d65b-4f99-ad24-5c1d443bff4b\") " pod="kube-system/global-pull-secret-syncer-9hl5q" Jul 13 22:13:28.802906 ip-10-0-133-179 kubenswrapper[2583]: E0713 22:13:28.802810 2583 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 13 22:13:28.803085 ip-10-0-133-179 kubenswrapper[2583]: E0713 22:13:28.802954 2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d38c850d-d65b-4f99-ad24-5c1d443bff4b-original-pull-secret podName:d38c850d-d65b-4f99-ad24-5c1d443bff4b nodeName:}" failed. No retries permitted until 2026-07-13 22:13:30.802937322 +0000 UTC m=+6.050852652 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/d38c850d-d65b-4f99-ad24-5c1d443bff4b-original-pull-secret") pod "global-pull-secret-syncer-9hl5q" (UID: "d38c850d-d65b-4f99-ad24-5c1d443bff4b") : object "kube-system"/"original-pull-secret" not registered Jul 13 22:13:28.903963 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:13:28.903921 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tcrhd\" (UniqueName: \"kubernetes.io/projected/c8bbc3a9-6b5c-485d-b7ef-ca4cdca462ce-kube-api-access-tcrhd\") pod \"network-check-target-v8mkx\" (UID: \"c8bbc3a9-6b5c-485d-b7ef-ca4cdca462ce\") " pod="openshift-network-diagnostics/network-check-target-v8mkx" Jul 13 22:13:28.904124 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:13:28.903971 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/85265522-e114-4c04-b595-a26fcae86884-metrics-certs\") pod \"network-metrics-daemon-26w5h\" (UID: \"85265522-e114-4c04-b595-a26fcae86884\") " pod="openshift-multus/network-metrics-daemon-26w5h" Jul 13 22:13:28.904124 ip-10-0-133-179 kubenswrapper[2583]: E0713 22:13:28.904106 2583 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 13 22:13:28.904222 ip-10-0-133-179 kubenswrapper[2583]: E0713 22:13:28.904170 2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/85265522-e114-4c04-b595-a26fcae86884-metrics-certs podName:85265522-e114-4c04-b595-a26fcae86884 nodeName:}" failed. No retries permitted until 2026-07-13 22:13:30.904150536 +0000 UTC m=+6.152065868 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/85265522-e114-4c04-b595-a26fcae86884-metrics-certs") pod "network-metrics-daemon-26w5h" (UID: "85265522-e114-4c04-b595-a26fcae86884") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 13 22:13:28.904767 ip-10-0-133-179 kubenswrapper[2583]: E0713 22:13:28.904622 2583 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 13 22:13:28.904767 ip-10-0-133-179 kubenswrapper[2583]: E0713 22:13:28.904648 2583 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 13 22:13:28.904767 ip-10-0-133-179 kubenswrapper[2583]: E0713 22:13:28.904660 2583 projected.go:196] Error preparing data for projected volume kube-api-access-tcrhd for pod openshift-network-diagnostics/network-check-target-v8mkx: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 13 22:13:28.904767 ip-10-0-133-179 kubenswrapper[2583]: E0713 22:13:28.904722 2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/c8bbc3a9-6b5c-485d-b7ef-ca4cdca462ce-kube-api-access-tcrhd podName:c8bbc3a9-6b5c-485d-b7ef-ca4cdca462ce nodeName:}" failed. No retries permitted until 2026-07-13 22:13:30.904697566 +0000 UTC m=+6.152612913 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-tcrhd" (UniqueName: "kubernetes.io/projected/c8bbc3a9-6b5c-485d-b7ef-ca4cdca462ce-kube-api-access-tcrhd") pod "network-check-target-v8mkx" (UID: "c8bbc3a9-6b5c-485d-b7ef-ca4cdca462ce") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 13 22:13:29.175781 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:13:29.175688 2583 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160" Jul 13 22:13:29.226591 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:13:29.226486 2583 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-12 22:08:27 +0000 UTC" deadline="2028-04-30 11:25:20.654844106 +0000 UTC" Jul 13 22:13:29.226591 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:13:29.226548 2583 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="15757h11m51.428300633s" Jul 13 22:13:29.388313 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:13:29.387422 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-v8mkx" Jul 13 22:13:29.388313 ip-10-0-133-179 kubenswrapper[2583]: E0713 22:13:29.387596 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-v8mkx" podUID="c8bbc3a9-6b5c-485d-b7ef-ca4cdca462ce" Jul 13 22:13:29.388313 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:13:29.388004 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-26w5h" Jul 13 22:13:29.388313 ip-10-0-133-179 kubenswrapper[2583]: E0713 22:13:29.388100 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-26w5h" podUID="85265522-e114-4c04-b595-a26fcae86884" Jul 13 22:13:29.388313 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:13:29.388175 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-9hl5q" Jul 13 22:13:29.388313 ip-10-0-133-179 kubenswrapper[2583]: E0713 22:13:29.388250 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-9hl5q" podUID="d38c850d-d65b-4f99-ad24-5c1d443bff4b" Jul 13 22:13:29.409246 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:13:29.409176 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-sht7p" event={"ID":"e05ae6ca-894a-4061-91ac-a0a6ed25d1b2","Type":"ContainerStarted","Data":"7e6c1d9d68ded658d0ef8e8334db1d1d13248acd5a7f14ab51eac984b4ed3d9c"} Jul 13 22:13:29.434279 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:13:29.434123 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-brfkn" event={"ID":"3357149c-54b6-4e57-93f0-2b9bf43ea301","Type":"ContainerStarted","Data":"b50df6302a11e5aa085e12c743a3f057066fdd2a6151b38ddd09eef5b41c6008"} Jul 13 22:13:29.443081 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:13:29.443045 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-nbdsh" event={"ID":"03f58cdb-4d8b-41a9-aa82-dff50f158f0c","Type":"ContainerStarted","Data":"8ecc0d34a0dd6f0fd5b1a2a389a9e3fd7d507edd91c2035deb2ab4bdb1b1e3a5"} Jul 13 22:13:29.452255 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:13:29.452182 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-133-179.ec2.internal" event={"ID":"120f6b7b6960a904bf802202dca0166e","Type":"ContainerStarted","Data":"725c1328c8dcd8dbbbb708ad29fa9b2a4e4d494388c512a39c5591bfb4ba645c"} Jul 13 22:13:29.454546 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:13:29.454471 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-2jc7k" event={"ID":"01c7e40b-ef15-4cf6-b5e7-e7129825d668","Type":"ContainerStarted","Data":"e9ba48e5b68010bbe988d2683939d9331d02701238fa16af3cd4f4849c165f3c"} Jul 13 22:13:29.471460 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:13:29.471425 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2r94b" event={"ID":"5b796097-01b1-4a5d-a10e-2cf14087d82e","Type":"ContainerStarted","Data":"1d7f5485cc6255a2fcbdba27b611d5765dfb03b396d360461d9cb7faec2c3e5a"} Jul 13 22:13:29.492447 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:13:29.492119 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-khllb" event={"ID":"19a018fb-4dd4-4a76-9305-8f70a789d38d","Type":"ContainerStarted","Data":"0dcfa99e8fc919f7af6f1a4205eaa2faa01f25c6ba847862d5a059a1aeabcdec"} Jul 13 22:13:29.497915 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:13:29.497838 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-r26w6" event={"ID":"54e8db97-3d72-4aa1-80bb-e56f10c7e474","Type":"ContainerStarted","Data":"29c71c08f25dc0727f0fa67246158400b06ded3a10394e7c2b9ed06b83ee1f3a"} Jul 13 22:13:29.509622 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:13:29.509210 2583 generic.go:356] "Generic (PLEG): container finished" podID="8bd03286da06035aeb817bb940fc3daa" containerID="b922ab0a192db90af9e5da11a5675c34059146b550e861c2589238e0cbd509de" exitCode=0 Jul 13 22:13:29.509622 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:13:29.509305 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-179.ec2.internal" event={"ID":"8bd03286da06035aeb817bb940fc3daa","Type":"ContainerDied","Data":"b922ab0a192db90af9e5da11a5675c34059146b550e861c2589238e0cbd509de"} Jul 13 22:13:29.522313 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:13:29.522274 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-grjf8" event={"ID":"e8e90a01-fd99-4a55-ba17-522966f4fabe","Type":"ContainerStarted","Data":"ed612688796d264dfc5c38af4870ae086f611d2e173557adb108846ab89f35d3"} Jul 13 22:13:29.524637 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:13:29.524587 2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-133-179.ec2.internal" podStartSLOduration=3.524551387 podStartE2EDuration="3.524551387s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-13 22:13:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 22:13:29.467617517 +0000 UTC m=+4.715532868" watchObservedRunningTime="2026-07-13 22:13:29.524551387 +0000 UTC m=+4.772466738" Jul 13 22:13:29.529057 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:13:29.528965 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-wgcj4" event={"ID":"ad16aa2f-aede-4d47-82d6-0ab7dda663d4","Type":"ContainerStarted","Data":"6e5700b5f2386246cd9e7689a6ec31795d9212ef16edf94f6d3209890f709561"} Jul 13 22:13:30.550178 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:13:30.550049 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-179.ec2.internal" event={"ID":"8bd03286da06035aeb817bb940fc3daa","Type":"ContainerStarted","Data":"97685efe8d633d534cf416df752366f9099e8b0b3a88a3f512fb6730a1656dd3"} Jul 13 22:13:30.822169 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:13:30.822080 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/d38c850d-d65b-4f99-ad24-5c1d443bff4b-original-pull-secret\") pod \"global-pull-secret-syncer-9hl5q\" (UID: \"d38c850d-d65b-4f99-ad24-5c1d443bff4b\") " pod="kube-system/global-pull-secret-syncer-9hl5q" Jul 13 22:13:30.822340 ip-10-0-133-179 kubenswrapper[2583]: E0713 22:13:30.822297 2583 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 13 22:13:30.822415 ip-10-0-133-179 kubenswrapper[2583]: E0713 22:13:30.822377 2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d38c850d-d65b-4f99-ad24-5c1d443bff4b-original-pull-secret podName:d38c850d-d65b-4f99-ad24-5c1d443bff4b nodeName:}" failed. No retries permitted until 2026-07-13 22:13:34.822356198 +0000 UTC m=+10.070271529 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/d38c850d-d65b-4f99-ad24-5c1d443bff4b-original-pull-secret") pod "global-pull-secret-syncer-9hl5q" (UID: "d38c850d-d65b-4f99-ad24-5c1d443bff4b") : object "kube-system"/"original-pull-secret" not registered Jul 13 22:13:30.922980 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:13:30.922941 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tcrhd\" (UniqueName: \"kubernetes.io/projected/c8bbc3a9-6b5c-485d-b7ef-ca4cdca462ce-kube-api-access-tcrhd\") pod \"network-check-target-v8mkx\" (UID: \"c8bbc3a9-6b5c-485d-b7ef-ca4cdca462ce\") " pod="openshift-network-diagnostics/network-check-target-v8mkx" Jul 13 22:13:30.923169 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:13:30.922993 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/85265522-e114-4c04-b595-a26fcae86884-metrics-certs\") pod \"network-metrics-daemon-26w5h\" (UID: \"85265522-e114-4c04-b595-a26fcae86884\") " pod="openshift-multus/network-metrics-daemon-26w5h" Jul 13 22:13:30.923169 ip-10-0-133-179 kubenswrapper[2583]: E0713 22:13:30.923125 2583 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 13 22:13:30.923282 ip-10-0-133-179 kubenswrapper[2583]: E0713 22:13:30.923186 2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/85265522-e114-4c04-b595-a26fcae86884-metrics-certs podName:85265522-e114-4c04-b595-a26fcae86884 nodeName:}" failed. No retries permitted until 2026-07-13 22:13:34.923167815 +0000 UTC m=+10.171083157 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/85265522-e114-4c04-b595-a26fcae86884-metrics-certs") pod "network-metrics-daemon-26w5h" (UID: "85265522-e114-4c04-b595-a26fcae86884") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 13 22:13:30.923656 ip-10-0-133-179 kubenswrapper[2583]: E0713 22:13:30.923630 2583 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 13 22:13:30.923656 ip-10-0-133-179 kubenswrapper[2583]: E0713 22:13:30.923655 2583 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 13 22:13:30.923819 ip-10-0-133-179 kubenswrapper[2583]: E0713 22:13:30.923667 2583 projected.go:196] Error preparing data for projected volume kube-api-access-tcrhd for pod openshift-network-diagnostics/network-check-target-v8mkx: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 13 22:13:30.923819 ip-10-0-133-179 kubenswrapper[2583]: E0713 22:13:30.923716 2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/c8bbc3a9-6b5c-485d-b7ef-ca4cdca462ce-kube-api-access-tcrhd podName:c8bbc3a9-6b5c-485d-b7ef-ca4cdca462ce nodeName:}" failed. No retries permitted until 2026-07-13 22:13:34.923700464 +0000 UTC m=+10.171615795 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-tcrhd" (UniqueName: "kubernetes.io/projected/c8bbc3a9-6b5c-485d-b7ef-ca4cdca462ce-kube-api-access-tcrhd") pod "network-check-target-v8mkx" (UID: "c8bbc3a9-6b5c-485d-b7ef-ca4cdca462ce") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 13 22:13:31.384707 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:13:31.384678 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-9hl5q" Jul 13 22:13:31.384913 ip-10-0-133-179 kubenswrapper[2583]: E0713 22:13:31.384807 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-9hl5q" podUID="d38c850d-d65b-4f99-ad24-5c1d443bff4b" Jul 13 22:13:31.385210 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:13:31.385186 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-v8mkx" Jul 13 22:13:31.385318 ip-10-0-133-179 kubenswrapper[2583]: E0713 22:13:31.385278 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-v8mkx" podUID="c8bbc3a9-6b5c-485d-b7ef-ca4cdca462ce" Jul 13 22:13:31.385470 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:13:31.385455 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-26w5h" Jul 13 22:13:31.385596 ip-10-0-133-179 kubenswrapper[2583]: E0713 22:13:31.385560 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-26w5h" podUID="85265522-e114-4c04-b595-a26fcae86884" Jul 13 22:13:33.384676 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:13:33.384585 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-v8mkx" Jul 13 22:13:33.384676 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:13:33.384621 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-9hl5q" Jul 13 22:13:33.385308 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:13:33.384587 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-26w5h" Jul 13 22:13:33.385308 ip-10-0-133-179 kubenswrapper[2583]: E0713 22:13:33.384726 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-v8mkx" podUID="c8bbc3a9-6b5c-485d-b7ef-ca4cdca462ce" Jul 13 22:13:33.385308 ip-10-0-133-179 kubenswrapper[2583]: E0713 22:13:33.384834 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-26w5h" podUID="85265522-e114-4c04-b595-a26fcae86884" Jul 13 22:13:33.385308 ip-10-0-133-179 kubenswrapper[2583]: E0713 22:13:33.384925 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-9hl5q" podUID="d38c850d-d65b-4f99-ad24-5c1d443bff4b" Jul 13 22:13:34.859100 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:13:34.859016 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/d38c850d-d65b-4f99-ad24-5c1d443bff4b-original-pull-secret\") pod \"global-pull-secret-syncer-9hl5q\" (UID: \"d38c850d-d65b-4f99-ad24-5c1d443bff4b\") " pod="kube-system/global-pull-secret-syncer-9hl5q" Jul 13 22:13:34.859541 ip-10-0-133-179 kubenswrapper[2583]: E0713 22:13:34.859158 2583 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 13 22:13:34.859541 ip-10-0-133-179 kubenswrapper[2583]: E0713 22:13:34.859228 2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d38c850d-d65b-4f99-ad24-5c1d443bff4b-original-pull-secret podName:d38c850d-d65b-4f99-ad24-5c1d443bff4b nodeName:}" failed. No retries permitted until 2026-07-13 22:13:42.859212962 +0000 UTC m=+18.107128290 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/d38c850d-d65b-4f99-ad24-5c1d443bff4b-original-pull-secret") pod "global-pull-secret-syncer-9hl5q" (UID: "d38c850d-d65b-4f99-ad24-5c1d443bff4b") : object "kube-system"/"original-pull-secret" not registered Jul 13 22:13:34.959664 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:13:34.959628 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tcrhd\" (UniqueName: \"kubernetes.io/projected/c8bbc3a9-6b5c-485d-b7ef-ca4cdca462ce-kube-api-access-tcrhd\") pod \"network-check-target-v8mkx\" (UID: \"c8bbc3a9-6b5c-485d-b7ef-ca4cdca462ce\") " pod="openshift-network-diagnostics/network-check-target-v8mkx" Jul 13 22:13:34.959841 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:13:34.959681 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/85265522-e114-4c04-b595-a26fcae86884-metrics-certs\") pod \"network-metrics-daemon-26w5h\" (UID: \"85265522-e114-4c04-b595-a26fcae86884\") " pod="openshift-multus/network-metrics-daemon-26w5h" Jul 13 22:13:34.959841 ip-10-0-133-179 kubenswrapper[2583]: E0713 22:13:34.959797 2583 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 13 22:13:34.959841 ip-10-0-133-179 kubenswrapper[2583]: E0713 22:13:34.959832 2583 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 13 22:13:34.959987 ip-10-0-133-179 kubenswrapper[2583]: E0713 22:13:34.959852 2583 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 13 22:13:34.959987 ip-10-0-133-179 kubenswrapper[2583]: E0713 22:13:34.959865 2583 projected.go:196] Error preparing data for projected volume kube-api-access-tcrhd for pod openshift-network-diagnostics/network-check-target-v8mkx: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 13 22:13:34.959987 ip-10-0-133-179 kubenswrapper[2583]: E0713 22:13:34.959880 2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/85265522-e114-4c04-b595-a26fcae86884-metrics-certs podName:85265522-e114-4c04-b595-a26fcae86884 nodeName:}" failed. No retries permitted until 2026-07-13 22:13:42.959859353 +0000 UTC m=+18.207774682 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/85265522-e114-4c04-b595-a26fcae86884-metrics-certs") pod "network-metrics-daemon-26w5h" (UID: "85265522-e114-4c04-b595-a26fcae86884") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 13 22:13:34.959987 ip-10-0-133-179 kubenswrapper[2583]: E0713 22:13:34.959915 2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/c8bbc3a9-6b5c-485d-b7ef-ca4cdca462ce-kube-api-access-tcrhd podName:c8bbc3a9-6b5c-485d-b7ef-ca4cdca462ce nodeName:}" failed. No retries permitted until 2026-07-13 22:13:42.959899916 +0000 UTC m=+18.207815269 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-tcrhd" (UniqueName: "kubernetes.io/projected/c8bbc3a9-6b5c-485d-b7ef-ca4cdca462ce-kube-api-access-tcrhd") pod "network-check-target-v8mkx" (UID: "c8bbc3a9-6b5c-485d-b7ef-ca4cdca462ce") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 13 22:13:35.385090 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:13:35.384916 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-v8mkx" Jul 13 22:13:35.385090 ip-10-0-133-179 kubenswrapper[2583]: E0713 22:13:35.385038 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-v8mkx" podUID="c8bbc3a9-6b5c-485d-b7ef-ca4cdca462ce" Jul 13 22:13:35.385310 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:13:35.385118 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-26w5h" Jul 13 22:13:35.385310 ip-10-0-133-179 kubenswrapper[2583]: E0713 22:13:35.385234 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-26w5h" podUID="85265522-e114-4c04-b595-a26fcae86884" Jul 13 22:13:35.385310 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:13:35.385118 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-9hl5q" Jul 13 22:13:35.385481 ip-10-0-133-179 kubenswrapper[2583]: E0713 22:13:35.385327 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-9hl5q" podUID="d38c850d-d65b-4f99-ad24-5c1d443bff4b" Jul 13 22:13:37.384039 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:13:37.384009 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-26w5h" Jul 13 22:13:37.384520 ip-10-0-133-179 kubenswrapper[2583]: E0713 22:13:37.384127 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-26w5h" podUID="85265522-e114-4c04-b595-a26fcae86884" Jul 13 22:13:37.384520 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:13:37.384149 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-9hl5q" Jul 13 22:13:37.384520 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:13:37.384009 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-v8mkx" Jul 13 22:13:37.384520 ip-10-0-133-179 kubenswrapper[2583]: E0713 22:13:37.384262 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-9hl5q" podUID="d38c850d-d65b-4f99-ad24-5c1d443bff4b" Jul 13 22:13:37.384520 ip-10-0-133-179 kubenswrapper[2583]: E0713 22:13:37.384349 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-v8mkx" podUID="c8bbc3a9-6b5c-485d-b7ef-ca4cdca462ce" Jul 13 22:13:39.383975 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:13:39.383876 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-v8mkx" Jul 13 22:13:39.383975 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:13:39.383876 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-26w5h" Jul 13 22:13:39.384436 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:13:39.383896 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-9hl5q" Jul 13 22:13:39.384436 ip-10-0-133-179 kubenswrapper[2583]: E0713 22:13:39.384013 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-v8mkx" podUID="c8bbc3a9-6b5c-485d-b7ef-ca4cdca462ce" Jul 13 22:13:39.384436 ip-10-0-133-179 kubenswrapper[2583]: E0713 22:13:39.384074 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-26w5h" podUID="85265522-e114-4c04-b595-a26fcae86884" Jul 13 22:13:39.384436 ip-10-0-133-179 kubenswrapper[2583]: E0713 22:13:39.384186 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-9hl5q" podUID="d38c850d-d65b-4f99-ad24-5c1d443bff4b" Jul 13 22:13:41.383782 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:13:41.383670 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-v8mkx" Jul 13 22:13:41.383782 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:13:41.383713 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-26w5h" Jul 13 22:13:41.384348 ip-10-0-133-179 kubenswrapper[2583]: E0713 22:13:41.383814 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-v8mkx" podUID="c8bbc3a9-6b5c-485d-b7ef-ca4cdca462ce" Jul 13 22:13:41.384348 ip-10-0-133-179 kubenswrapper[2583]: E0713 22:13:41.383946 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-26w5h" podUID="85265522-e114-4c04-b595-a26fcae86884" Jul 13 22:13:41.384348 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:13:41.384017 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-9hl5q" Jul 13 22:13:41.384348 ip-10-0-133-179 kubenswrapper[2583]: E0713 22:13:41.384122 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-9hl5q" podUID="d38c850d-d65b-4f99-ad24-5c1d443bff4b" Jul 13 22:13:42.917011 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:13:42.916973 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/d38c850d-d65b-4f99-ad24-5c1d443bff4b-original-pull-secret\") pod \"global-pull-secret-syncer-9hl5q\" (UID: \"d38c850d-d65b-4f99-ad24-5c1d443bff4b\") " pod="kube-system/global-pull-secret-syncer-9hl5q" Jul 13 22:13:42.917633 ip-10-0-133-179 kubenswrapper[2583]: E0713 22:13:42.917109 2583 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 13 22:13:42.917633 ip-10-0-133-179 kubenswrapper[2583]: E0713 22:13:42.917185 2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d38c850d-d65b-4f99-ad24-5c1d443bff4b-original-pull-secret podName:d38c850d-d65b-4f99-ad24-5c1d443bff4b nodeName:}" failed. No retries permitted until 2026-07-13 22:13:58.917164181 +0000 UTC m=+34.165079521 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/d38c850d-d65b-4f99-ad24-5c1d443bff4b-original-pull-secret") pod "global-pull-secret-syncer-9hl5q" (UID: "d38c850d-d65b-4f99-ad24-5c1d443bff4b") : object "kube-system"/"original-pull-secret" not registered Jul 13 22:13:43.017982 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:13:43.017942 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tcrhd\" (UniqueName: \"kubernetes.io/projected/c8bbc3a9-6b5c-485d-b7ef-ca4cdca462ce-kube-api-access-tcrhd\") pod \"network-check-target-v8mkx\" (UID: \"c8bbc3a9-6b5c-485d-b7ef-ca4cdca462ce\") " pod="openshift-network-diagnostics/network-check-target-v8mkx" Jul 13 22:13:43.017982 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:13:43.017986 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/85265522-e114-4c04-b595-a26fcae86884-metrics-certs\") pod \"network-metrics-daemon-26w5h\" (UID: \"85265522-e114-4c04-b595-a26fcae86884\") " pod="openshift-multus/network-metrics-daemon-26w5h" Jul 13 22:13:43.018267 ip-10-0-133-179 kubenswrapper[2583]: E0713 22:13:43.018116 2583 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 13 22:13:43.018267 ip-10-0-133-179 kubenswrapper[2583]: E0713 22:13:43.018144 2583 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 13 22:13:43.018267 ip-10-0-133-179 kubenswrapper[2583]: E0713 22:13:43.018158 2583 projected.go:196] Error preparing data for projected volume kube-api-access-tcrhd for pod openshift-network-diagnostics/network-check-target-v8mkx: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 13 22:13:43.018267 ip-10-0-133-179 kubenswrapper[2583]: E0713 22:13:43.018170 2583 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 13 22:13:43.018267 ip-10-0-133-179 kubenswrapper[2583]: E0713 22:13:43.018223 2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/c8bbc3a9-6b5c-485d-b7ef-ca4cdca462ce-kube-api-access-tcrhd podName:c8bbc3a9-6b5c-485d-b7ef-ca4cdca462ce nodeName:}" failed. No retries permitted until 2026-07-13 22:13:59.018204546 +0000 UTC m=+34.266119874 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-tcrhd" (UniqueName: "kubernetes.io/projected/c8bbc3a9-6b5c-485d-b7ef-ca4cdca462ce-kube-api-access-tcrhd") pod "network-check-target-v8mkx" (UID: "c8bbc3a9-6b5c-485d-b7ef-ca4cdca462ce") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 13 22:13:43.018267 ip-10-0-133-179 kubenswrapper[2583]: E0713 22:13:43.018245 2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/85265522-e114-4c04-b595-a26fcae86884-metrics-certs podName:85265522-e114-4c04-b595-a26fcae86884 nodeName:}" failed. No retries permitted until 2026-07-13 22:13:59.018234604 +0000 UTC m=+34.266149935 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/85265522-e114-4c04-b595-a26fcae86884-metrics-certs") pod "network-metrics-daemon-26w5h" (UID: "85265522-e114-4c04-b595-a26fcae86884") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 13 22:13:43.383936 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:13:43.383849 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-9hl5q" Jul 13 22:13:43.384095 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:13:43.383970 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-26w5h" Jul 13 22:13:43.384095 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:13:43.383982 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-v8mkx" Jul 13 22:13:43.384095 ip-10-0-133-179 kubenswrapper[2583]: E0713 22:13:43.383969 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-9hl5q" podUID="d38c850d-d65b-4f99-ad24-5c1d443bff4b" Jul 13 22:13:43.384095 ip-10-0-133-179 kubenswrapper[2583]: E0713 22:13:43.384084 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-26w5h" podUID="85265522-e114-4c04-b595-a26fcae86884" Jul 13 22:13:43.384315 ip-10-0-133-179 kubenswrapper[2583]: E0713 22:13:43.384159 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-v8mkx" podUID="c8bbc3a9-6b5c-485d-b7ef-ca4cdca462ce" Jul 13 22:13:45.385395 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:13:45.385362 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-v8mkx" Jul 13 22:13:45.385863 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:13:45.385477 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-9hl5q" Jul 13 22:13:45.385863 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:13:45.385493 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-26w5h" Jul 13 22:13:45.385863 ip-10-0-133-179 kubenswrapper[2583]: E0713 22:13:45.385503 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-v8mkx" podUID="c8bbc3a9-6b5c-485d-b7ef-ca4cdca462ce" Jul 13 22:13:45.385863 ip-10-0-133-179 kubenswrapper[2583]: E0713 22:13:45.385585 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-9hl5q" podUID="d38c850d-d65b-4f99-ad24-5c1d443bff4b" Jul 13 22:13:45.385863 ip-10-0-133-179 kubenswrapper[2583]: E0713 22:13:45.385709 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-26w5h" podUID="85265522-e114-4c04-b595-a26fcae86884" Jul 13 22:13:47.385873 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:13:47.385841 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-v8mkx" Jul 13 22:13:47.386240 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:13:47.385904 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-9hl5q" Jul 13 22:13:47.386240 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:13:47.385921 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-26w5h" Jul 13 22:13:47.386240 ip-10-0-133-179 kubenswrapper[2583]: E0713 22:13:47.386041 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-26w5h" podUID="85265522-e114-4c04-b595-a26fcae86884" Jul 13 22:13:47.386240 ip-10-0-133-179 kubenswrapper[2583]: E0713 22:13:47.386134 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-9hl5q" podUID="d38c850d-d65b-4f99-ad24-5c1d443bff4b" Jul 13 22:13:47.386240 ip-10-0-133-179 kubenswrapper[2583]: E0713 22:13:47.386199 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-v8mkx" podUID="c8bbc3a9-6b5c-485d-b7ef-ca4cdca462ce" Jul 13 22:13:47.582723 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:13:47.582376 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-2jc7k" event={"ID":"01c7e40b-ef15-4cf6-b5e7-e7129825d668","Type":"ContainerStarted","Data":"cb3031f6155547efc829673a3091730530655be72770d23a5efa7406c2e3f3e5"} Jul 13 22:13:47.584029 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:13:47.584004 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2r94b" event={"ID":"5b796097-01b1-4a5d-a10e-2cf14087d82e","Type":"ContainerStarted","Data":"9e1be6f7ea6c4fbdfd75ef5f927b37fe8955c3c116e5f9c94f08b384b09b9ca9"} Jul 13 22:13:47.585738 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:13:47.585706 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-khllb" event={"ID":"19a018fb-4dd4-4a76-9305-8f70a789d38d","Type":"ContainerStarted","Data":"405ccf1318a99208b21b7e7bf0e67861b7013599cc3a35888d25aa0dc4691516"} Jul 13 22:13:47.587368 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:13:47.587211 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-r26w6" event={"ID":"54e8db97-3d72-4aa1-80bb-e56f10c7e474","Type":"ContainerStarted","Data":"a3056c1fc8ecce993b716bc3b77e8a8480f6e7b142d4a6866c8ad97756657303"} Jul 13 22:13:47.588627 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:13:47.588602 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-grjf8" event={"ID":"e8e90a01-fd99-4a55-ba17-522966f4fabe","Type":"ContainerStarted","Data":"c5b149790990ec0f235cdf779b3971deb0bc285298970232cac0aa506a9a065c"} Jul 13 22:13:47.589843 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:13:47.589815 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-wgcj4" event={"ID":"ad16aa2f-aede-4d47-82d6-0ab7dda663d4","Type":"ContainerStarted","Data":"339cb2ad461ac2e8eb647781460bc3cd0c12de5ef20d5541e2c192d295f5d346"} Jul 13 22:13:47.591014 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:13:47.590995 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-sht7p" event={"ID":"e05ae6ca-894a-4061-91ac-a0a6ed25d1b2","Type":"ContainerStarted","Data":"cf9945e9fe9594d27b5684417a8e0b04150b8e90d1b429529176b9d53ce4386f"} Jul 13 22:13:47.592217 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:13:47.592200 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-nbdsh" event={"ID":"03f58cdb-4d8b-41a9-aa82-dff50f158f0c","Type":"ContainerStarted","Data":"07a799be0a16241df59254bb645c62258241073779453939fb5ec9941f69f931"} Jul 13 22:13:47.600332 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:13:47.600294 2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-179.ec2.internal" podStartSLOduration=21.600281804 podStartE2EDuration="21.600281804s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-13 22:13:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 22:13:30.569698613 +0000 UTC m=+5.817613964" watchObservedRunningTime="2026-07-13 22:13:47.600281804 +0000 UTC m=+22.848197154" Jul 13 22:13:47.600755 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:13:47.600730 2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-2jc7k" podStartSLOduration=3.679534905 podStartE2EDuration="22.600720841s" totalImagesPullingTime="18.921185936s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-13 22:13:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 22:13:47.600150537 +0000 UTC m=+22.848065887" watchObservedRunningTime="2026-07-13 22:13:47.600720841 +0000 UTC m=+22.848636191" Jul 13 22:13:47.646051 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:13:47.645983 2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-sht7p" podStartSLOduration=3.785420836 podStartE2EDuration="22.645962601s" totalImagesPullingTime="18.860541765s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-13 22:13:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 22:13:47.645626289 +0000 UTC m=+22.893541640" watchObservedRunningTime="2026-07-13 22:13:47.645962601 +0000 UTC m=+22.893877952" Jul 13 22:13:47.662935 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:13:47.662759 2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-grjf8" podStartSLOduration=3.802936862 podStartE2EDuration="22.662743482s" totalImagesPullingTime="18.85980662s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-13 22:13:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 22:13:47.662521323 +0000 UTC m=+22.910436672" watchObservedRunningTime="2026-07-13 22:13:47.662743482 +0000 UTC m=+22.910658832" Jul 13 22:13:47.682783 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:13:47.682729 2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-r26w6" podStartSLOduration=3.797619895 podStartE2EDuration="22.682707449s" totalImagesPullingTime="18.885087554s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-13 22:13:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 22:13:47.682099631 +0000 UTC m=+22.930014981" watchObservedRunningTime="2026-07-13 22:13:47.682707449 +0000 UTC m=+22.930622801" Jul 13 22:13:47.704920 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:13:47.704862 2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-wgcj4" podStartSLOduration=12.029062399 podStartE2EDuration="22.704840579s" totalImagesPullingTime="10.67577818s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-13 22:13:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 22:13:47.704640771 +0000 UTC m=+22.952556121" watchObservedRunningTime="2026-07-13 22:13:47.704840579 +0000 UTC m=+22.952755930" Jul 13 22:13:48.596331 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:13:48.596238 2583 generic.go:356] "Generic (PLEG): container finished" podID="03f58cdb-4d8b-41a9-aa82-dff50f158f0c" containerID="07a799be0a16241df59254bb645c62258241073779453939fb5ec9941f69f931" exitCode=0 Jul 13 22:13:48.597040 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:13:48.596337 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-nbdsh" event={"ID":"03f58cdb-4d8b-41a9-aa82-dff50f158f0c","Type":"ContainerDied","Data":"07a799be0a16241df59254bb645c62258241073779453939fb5ec9941f69f931"} Jul 13 22:13:48.599381 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:13:48.599349 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-khllb" event={"ID":"19a018fb-4dd4-4a76-9305-8f70a789d38d","Type":"ContainerStarted","Data":"32edb23dc8415be638452bbf584620e6d81310c45e6d995b1c0611f05aaf535e"} Jul 13 22:13:48.599494 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:13:48.599434 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-khllb" event={"ID":"19a018fb-4dd4-4a76-9305-8f70a789d38d","Type":"ContainerStarted","Data":"efff77f23f2d246dfa986ece9cd65eeaf808a314b402862f0b39a3144fbef1bf"} Jul 13 22:13:48.599494 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:13:48.599486 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-khllb" event={"ID":"19a018fb-4dd4-4a76-9305-8f70a789d38d","Type":"ContainerStarted","Data":"a2a0bacd8c6fc9179bc565cd2443bcaf79bb19d41e982eaba536853328f9d3e2"} Jul 13 22:13:48.599618 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:13:48.599532 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-khllb" event={"ID":"19a018fb-4dd4-4a76-9305-8f70a789d38d","Type":"ContainerStarted","Data":"63ecb9bdd2ba76f13a5279d2042fc40e86e8348df41c90d88dab350d4dabfdb1"} Jul 13 22:13:48.599618 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:13:48.599606 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-khllb" event={"ID":"19a018fb-4dd4-4a76-9305-8f70a789d38d","Type":"ContainerStarted","Data":"0ff88719c865f91d9f39caf19087c3c810f89270e185b4c3fefdf5d73b363f9a"} Jul 13 22:13:48.600923 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:13:48.600897 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-brfkn" event={"ID":"3357149c-54b6-4e57-93f0-2b9bf43ea301","Type":"ContainerStarted","Data":"697b08d7fea47a6cfa641bb2cb53e5121c503bdae93d8d5524684839640b401d"} Jul 13 22:13:48.876563 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:13:48.876531 2583 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock" Jul 13 22:13:49.261325 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:13:49.261157 2583 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-07-13T22:13:48.876557669Z","UUID":"b1fe8399-40b6-4808-8215-eb50b581fbcc","Handler":null,"Name":"","Endpoint":""} Jul 13 22:13:49.262987 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:13:49.262951 2583 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0 Jul 13 22:13:49.263098 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:13:49.262999 2583 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock Jul 13 22:13:49.383824 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:13:49.383802 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-26w5h" Jul 13 22:13:49.384090 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:13:49.383797 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-v8mkx" Jul 13 22:13:49.384090 ip-10-0-133-179 kubenswrapper[2583]: E0713 22:13:49.383922 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-26w5h" podUID="85265522-e114-4c04-b595-a26fcae86884" Jul 13 22:13:49.384090 ip-10-0-133-179 kubenswrapper[2583]: E0713 22:13:49.384020 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-v8mkx" podUID="c8bbc3a9-6b5c-485d-b7ef-ca4cdca462ce" Jul 13 22:13:49.384263 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:13:49.384100 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-9hl5q" Jul 13 22:13:49.384263 ip-10-0-133-179 kubenswrapper[2583]: E0713 22:13:49.384182 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-9hl5q" podUID="d38c850d-d65b-4f99-ad24-5c1d443bff4b" Jul 13 22:13:49.605692 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:13:49.605601 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2r94b" event={"ID":"5b796097-01b1-4a5d-a10e-2cf14087d82e","Type":"ContainerStarted","Data":"80c280228797d198ac3365072db432799a812a6b4a21459703247994fd0d775f"} Jul 13 22:13:50.609230 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:13:50.609137 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2r94b" event={"ID":"5b796097-01b1-4a5d-a10e-2cf14087d82e","Type":"ContainerStarted","Data":"0eda4a354a0649a8c2652d338d0394ba2c157284c70e8cac7c1dc3b47615eb17"} Jul 13 22:13:50.612191 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:13:50.612149 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-khllb" event={"ID":"19a018fb-4dd4-4a76-9305-8f70a789d38d","Type":"ContainerStarted","Data":"3bdb1ba8f886d2c3dbb358bd19ab548b3eafb1a1f9b65bac451a5a9b8b2da4b5"} Jul 13 22:13:50.632915 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:13:50.632855 2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2r94b" podStartSLOduration=4.19746852 podStartE2EDuration="25.632838876s" totalImagesPullingTime="21.435370356s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-13 22:13:25 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 22:13:50.632719027 +0000 UTC m=+25.880634378" watchObservedRunningTime="2026-07-13 22:13:50.632838876 +0000 UTC m=+25.880754227" Jul 13 22:13:50.633275 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:13:50.633251 2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-brfkn" podStartSLOduration=6.772751925 podStartE2EDuration="25.633244694s" totalImagesPullingTime="18.860492769s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-13 22:13:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 22:13:48.634483765 +0000 UTC m=+23.882399126" watchObservedRunningTime="2026-07-13 22:13:50.633244694 +0000 UTC m=+25.881160043" Jul 13 22:13:51.185548 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:13:51.185510 2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-wgcj4" Jul 13 22:13:51.186166 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:13:51.186146 2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-wgcj4" Jul 13 22:13:51.384339 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:13:51.384305 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-v8mkx" Jul 13 22:13:51.384339 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:13:51.384340 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-9hl5q" Jul 13 22:13:51.384543 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:13:51.384312 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-26w5h" Jul 13 22:13:51.384543 ip-10-0-133-179 kubenswrapper[2583]: E0713 22:13:51.384407 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-v8mkx" podUID="c8bbc3a9-6b5c-485d-b7ef-ca4cdca462ce" Jul 13 22:13:51.387383 ip-10-0-133-179 kubenswrapper[2583]: E0713 22:13:51.384945 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-26w5h" podUID="85265522-e114-4c04-b595-a26fcae86884" Jul 13 22:13:51.387383 ip-10-0-133-179 kubenswrapper[2583]: E0713 22:13:51.385330 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-9hl5q" podUID="d38c850d-d65b-4f99-ad24-5c1d443bff4b" Jul 13 22:13:51.614557 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:13:51.614483 2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-wgcj4" Jul 13 22:13:51.614986 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:13:51.614971 2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-wgcj4" Jul 13 22:13:53.384209 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:13:53.383866 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-9hl5q" Jul 13 22:13:53.384846 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:13:53.383871 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-26w5h" Jul 13 22:13:53.384846 ip-10-0-133-179 kubenswrapper[2583]: E0713 22:13:53.384268 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-9hl5q" podUID="d38c850d-d65b-4f99-ad24-5c1d443bff4b" Jul 13 22:13:53.384846 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:13:53.383926 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-v8mkx" Jul 13 22:13:53.384846 ip-10-0-133-179 kubenswrapper[2583]: E0713 22:13:53.384335 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-26w5h" podUID="85265522-e114-4c04-b595-a26fcae86884" Jul 13 22:13:53.384846 ip-10-0-133-179 kubenswrapper[2583]: E0713 22:13:53.384421 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-v8mkx" podUID="c8bbc3a9-6b5c-485d-b7ef-ca4cdca462ce" Jul 13 22:13:53.620072 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:13:53.620038 2583 generic.go:356] "Generic (PLEG): container finished" podID="03f58cdb-4d8b-41a9-aa82-dff50f158f0c" containerID="ce8c9ed77c3ad2846a18eea5e18488e59da2e7a079a32cd6512d986ad2a5a987" exitCode=0 Jul 13 22:13:53.620255 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:13:53.620125 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-nbdsh" event={"ID":"03f58cdb-4d8b-41a9-aa82-dff50f158f0c","Type":"ContainerDied","Data":"ce8c9ed77c3ad2846a18eea5e18488e59da2e7a079a32cd6512d986ad2a5a987"} Jul 13 22:13:53.623807 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:13:53.623747 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-khllb" event={"ID":"19a018fb-4dd4-4a76-9305-8f70a789d38d","Type":"ContainerStarted","Data":"22241f64b7dba92db39e450bc478bc532c9bf90151944856ea196fe650198f8f"} Jul 13 22:13:53.624007 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:13:53.623980 2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-khllb" Jul 13 22:13:53.624121 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:13:53.624052 2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-khllb" Jul 13 22:13:53.624121 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:13:53.624084 2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-khllb" Jul 13 22:13:53.639141 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:13:53.639051 2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-khllb" Jul 13 22:13:53.639248 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:13:53.639203 2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-khllb" Jul 13 22:13:53.668487 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:13:53.668436 2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-khllb" podStartSLOduration=9.720316541999999 podStartE2EDuration="28.668417863s" totalImagesPullingTime="18.948101321s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-13 22:13:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 22:13:53.666972254 +0000 UTC m=+28.914887603" watchObservedRunningTime="2026-07-13 22:13:53.668417863 +0000 UTC m=+28.916333212" Jul 13 22:13:55.068452 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:13:55.068423 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-9hl5q"] Jul 13 22:13:55.068965 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:13:55.068604 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-9hl5q" Jul 13 22:13:55.068965 ip-10-0-133-179 kubenswrapper[2583]: E0713 22:13:55.068715 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-9hl5q" podUID="d38c850d-d65b-4f99-ad24-5c1d443bff4b" Jul 13 22:13:55.073997 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:13:55.073960 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-26w5h"] Jul 13 22:13:55.074118 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:13:55.074084 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-26w5h" Jul 13 22:13:55.074233 ip-10-0-133-179 kubenswrapper[2583]: E0713 22:13:55.074212 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-26w5h" podUID="85265522-e114-4c04-b595-a26fcae86884" Jul 13 22:13:55.074675 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:13:55.074654 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-v8mkx"] Jul 13 22:13:55.074777 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:13:55.074731 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-v8mkx" Jul 13 22:13:55.074839 ip-10-0-133-179 kubenswrapper[2583]: E0713 22:13:55.074798 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-v8mkx" podUID="c8bbc3a9-6b5c-485d-b7ef-ca4cdca462ce" Jul 13 22:13:56.383797 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:13:56.383765 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-9hl5q" Jul 13 22:13:56.384199 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:13:56.383865 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-v8mkx" Jul 13 22:13:56.384199 ip-10-0-133-179 kubenswrapper[2583]: E0713 22:13:56.383871 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-9hl5q" podUID="d38c850d-d65b-4f99-ad24-5c1d443bff4b" Jul 13 22:13:56.384199 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:13:56.383893 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-26w5h" Jul 13 22:13:56.384199 ip-10-0-133-179 kubenswrapper[2583]: E0713 22:13:56.383973 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-v8mkx" podUID="c8bbc3a9-6b5c-485d-b7ef-ca4cdca462ce" Jul 13 22:13:56.384199 ip-10-0-133-179 kubenswrapper[2583]: E0713 22:13:56.384041 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-26w5h" podUID="85265522-e114-4c04-b595-a26fcae86884" Jul 13 22:13:56.631951 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:13:56.631919 2583 generic.go:356] "Generic (PLEG): container finished" podID="03f58cdb-4d8b-41a9-aa82-dff50f158f0c" containerID="d34ce3bf0d2dac1e8e43afc0cfd2cab8ab206745a17b05226d9a87c33d54c3fa" exitCode=0 Jul 13 22:13:56.631951 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:13:56.631954 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-nbdsh" event={"ID":"03f58cdb-4d8b-41a9-aa82-dff50f158f0c","Type":"ContainerDied","Data":"d34ce3bf0d2dac1e8e43afc0cfd2cab8ab206745a17b05226d9a87c33d54c3fa"} Jul 13 22:13:58.384427 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:13:58.384255 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-9hl5q" Jul 13 22:13:58.384816 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:13:58.384264 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-v8mkx" Jul 13 22:13:58.384816 ip-10-0-133-179 kubenswrapper[2583]: E0713 22:13:58.384512 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-9hl5q" podUID="d38c850d-d65b-4f99-ad24-5c1d443bff4b" Jul 13 22:13:58.384816 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:13:58.384278 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-26w5h" Jul 13 22:13:58.384816 ip-10-0-133-179 kubenswrapper[2583]: E0713 22:13:58.384610 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-v8mkx" podUID="c8bbc3a9-6b5c-485d-b7ef-ca4cdca462ce" Jul 13 22:13:58.384816 ip-10-0-133-179 kubenswrapper[2583]: E0713 22:13:58.384689 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-26w5h" podUID="85265522-e114-4c04-b595-a26fcae86884" Jul 13 22:13:58.638796 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:13:58.638717 2583 generic.go:356] "Generic (PLEG): container finished" podID="03f58cdb-4d8b-41a9-aa82-dff50f158f0c" containerID="bb6a116b41f8fc8860658ff5e9ad0a60d3ef0c6e4bd6110fd26483b1607358ac" exitCode=0 Jul 13 22:13:58.638796 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:13:58.638769 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-nbdsh" event={"ID":"03f58cdb-4d8b-41a9-aa82-dff50f158f0c","Type":"ContainerDied","Data":"bb6a116b41f8fc8860658ff5e9ad0a60d3ef0c6e4bd6110fd26483b1607358ac"} Jul 13 22:13:58.940045 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:13:58.939968 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/d38c850d-d65b-4f99-ad24-5c1d443bff4b-original-pull-secret\") pod \"global-pull-secret-syncer-9hl5q\" (UID: \"d38c850d-d65b-4f99-ad24-5c1d443bff4b\") " pod="kube-system/global-pull-secret-syncer-9hl5q" Jul 13 22:13:58.940174 ip-10-0-133-179 kubenswrapper[2583]: E0713 22:13:58.940127 2583 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 13 22:13:58.940209 ip-10-0-133-179 kubenswrapper[2583]: E0713 22:13:58.940191 2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d38c850d-d65b-4f99-ad24-5c1d443bff4b-original-pull-secret podName:d38c850d-d65b-4f99-ad24-5c1d443bff4b nodeName:}" failed. No retries permitted until 2026-07-13 22:14:30.940171646 +0000 UTC m=+66.188086990 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/d38c850d-d65b-4f99-ad24-5c1d443bff4b-original-pull-secret") pod "global-pull-secret-syncer-9hl5q" (UID: "d38c850d-d65b-4f99-ad24-5c1d443bff4b") : object "kube-system"/"original-pull-secret" not registered Jul 13 22:13:59.040702 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:13:59.040672 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tcrhd\" (UniqueName: \"kubernetes.io/projected/c8bbc3a9-6b5c-485d-b7ef-ca4cdca462ce-kube-api-access-tcrhd\") pod \"network-check-target-v8mkx\" (UID: \"c8bbc3a9-6b5c-485d-b7ef-ca4cdca462ce\") " pod="openshift-network-diagnostics/network-check-target-v8mkx" Jul 13 22:13:59.040895 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:13:59.040710 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/85265522-e114-4c04-b595-a26fcae86884-metrics-certs\") pod \"network-metrics-daemon-26w5h\" (UID: \"85265522-e114-4c04-b595-a26fcae86884\") " pod="openshift-multus/network-metrics-daemon-26w5h" Jul 13 22:13:59.040895 ip-10-0-133-179 kubenswrapper[2583]: E0713 22:13:59.040813 2583 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 13 22:13:59.040895 ip-10-0-133-179 kubenswrapper[2583]: E0713 22:13:59.040818 2583 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 13 22:13:59.040895 ip-10-0-133-179 kubenswrapper[2583]: E0713 22:13:59.040845 2583 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 13 22:13:59.040895 ip-10-0-133-179 kubenswrapper[2583]: E0713 22:13:59.040858 2583 projected.go:196] Error preparing data for projected volume kube-api-access-tcrhd for pod openshift-network-diagnostics/network-check-target-v8mkx: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 13 22:13:59.040895 ip-10-0-133-179 kubenswrapper[2583]: E0713 22:13:59.040875 2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/85265522-e114-4c04-b595-a26fcae86884-metrics-certs podName:85265522-e114-4c04-b595-a26fcae86884 nodeName:}" failed. No retries permitted until 2026-07-13 22:14:31.040856427 +0000 UTC m=+66.288771775 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/85265522-e114-4c04-b595-a26fcae86884-metrics-certs") pod "network-metrics-daemon-26w5h" (UID: "85265522-e114-4c04-b595-a26fcae86884") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 13 22:13:59.041177 ip-10-0-133-179 kubenswrapper[2583]: E0713 22:13:59.040912 2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/c8bbc3a9-6b5c-485d-b7ef-ca4cdca462ce-kube-api-access-tcrhd podName:c8bbc3a9-6b5c-485d-b7ef-ca4cdca462ce nodeName:}" failed. No retries permitted until 2026-07-13 22:14:31.040898498 +0000 UTC m=+66.288813829 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-tcrhd" (UniqueName: "kubernetes.io/projected/c8bbc3a9-6b5c-485d-b7ef-ca4cdca462ce-kube-api-access-tcrhd") pod "network-check-target-v8mkx" (UID: "c8bbc3a9-6b5c-485d-b7ef-ca4cdca462ce") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 13 22:14:00.046609 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:14:00.046564 2583 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-179.ec2.internal" event="NodeReady" Jul 13 22:14:00.047005 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:14:00.046738 2583 kubelet_node_status.go:489] "Fast updating node status as it just became ready" Jul 13 22:14:00.100156 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:14:00.100067 2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-74f88c6495-7kk5p"] Jul 13 22:14:00.102284 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:14:00.102256 2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-w4tml"] Jul 13 22:14:00.102415 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:14:00.102382 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-74f88c6495-7kk5p" Jul 13 22:14:00.104334 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:14:00.104310 2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-sbw7w"] Jul 13 22:14:00.104523 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:14:00.104396 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-w4tml" Jul 13 22:14:00.104871 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:14:00.104853 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-zb4qz\"" Jul 13 22:14:00.104960 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:14:00.104853 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\"" Jul 13 22:14:00.104960 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:14:00.104856 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\"" Jul 13 22:14:00.104960 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:14:00.104856 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\"" Jul 13 22:14:00.106226 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:14:00.106209 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-sbw7w" Jul 13 22:14:00.106845 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:14:00.106796 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-p77sr\"" Jul 13 22:14:00.106845 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:14:00.106808 2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\"" Jul 13 22:14:00.106845 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:14:00.106832 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\"" Jul 13 22:14:00.109334 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:14:00.109313 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-9d5hg\"" Jul 13 22:14:00.109451 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:14:00.109405 2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\"" Jul 13 22:14:00.110258 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:14:00.110136 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\"" Jul 13 22:14:00.110258 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:14:00.110146 2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\"" Jul 13 22:14:00.111828 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:14:00.111807 2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\"" Jul 13 22:14:00.114903 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:14:00.114882 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-sbw7w"] Jul 13 22:14:00.117229 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:14:00.117208 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-74f88c6495-7kk5p"] Jul 13 22:14:00.130492 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:14:00.130464 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-w4tml"] Jul 13 22:14:00.250775 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:14:00.250746 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e92b34db-2297-4fc3-baab-7babcc0a94ff-tmp-dir\") pod \"dns-default-w4tml\" (UID: \"e92b34db-2297-4fc3-baab-7babcc0a94ff\") " pod="openshift-dns/dns-default-w4tml" Jul 13 22:14:00.250775 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:14:00.250776 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/2690217c-291d-4cbb-8ede-0ebe7802731c-installation-pull-secrets\") pod \"image-registry-74f88c6495-7kk5p\" (UID: \"2690217c-291d-4cbb-8ede-0ebe7802731c\") " pod="openshift-image-registry/image-registry-74f88c6495-7kk5p" Jul 13 22:14:00.251015 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:14:00.250829 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/e92b34db-2297-4fc3-baab-7babcc0a94ff-metrics-tls\") pod \"dns-default-w4tml\" (UID: \"e92b34db-2297-4fc3-baab-7babcc0a94ff\") " pod="openshift-dns/dns-default-w4tml" Jul 13 22:14:00.251015 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:14:00.250877 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/2690217c-291d-4cbb-8ede-0ebe7802731c-registry-tls\") pod \"image-registry-74f88c6495-7kk5p\" (UID: \"2690217c-291d-4cbb-8ede-0ebe7802731c\") " pod="openshift-image-registry/image-registry-74f88c6495-7kk5p" Jul 13 22:14:00.251015 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:14:00.250915 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/2690217c-291d-4cbb-8ede-0ebe7802731c-trusted-ca\") pod \"image-registry-74f88c6495-7kk5p\" (UID: \"2690217c-291d-4cbb-8ede-0ebe7802731c\") " pod="openshift-image-registry/image-registry-74f88c6495-7kk5p" Jul 13 22:14:00.251015 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:14:00.250948 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rjkqx\" (UniqueName: \"kubernetes.io/projected/f8f9da95-14d8-4f4f-8142-0a2f347ba336-kube-api-access-rjkqx\") pod \"ingress-canary-sbw7w\" (UID: \"f8f9da95-14d8-4f4f-8142-0a2f347ba336\") " pod="openshift-ingress-canary/ingress-canary-sbw7w" Jul 13 22:14:00.251015 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:14:00.250987 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6lwmw\" (UniqueName: \"kubernetes.io/projected/2690217c-291d-4cbb-8ede-0ebe7802731c-kube-api-access-6lwmw\") pod \"image-registry-74f88c6495-7kk5p\" (UID: \"2690217c-291d-4cbb-8ede-0ebe7802731c\") " pod="openshift-image-registry/image-registry-74f88c6495-7kk5p" Jul 13 22:14:00.251262 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:14:00.251016 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/2690217c-291d-4cbb-8ede-0ebe7802731c-image-registry-private-configuration\") pod \"image-registry-74f88c6495-7kk5p\" (UID: \"2690217c-291d-4cbb-8ede-0ebe7802731c\") " pod="openshift-image-registry/image-registry-74f88c6495-7kk5p" Jul 13 22:14:00.251262 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:14:00.251034 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/f8f9da95-14d8-4f4f-8142-0a2f347ba336-cert\") pod \"ingress-canary-sbw7w\" (UID: \"f8f9da95-14d8-4f4f-8142-0a2f347ba336\") " pod="openshift-ingress-canary/ingress-canary-sbw7w" Jul 13 22:14:00.251262 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:14:00.251054 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m5zrs\" (UniqueName: \"kubernetes.io/projected/e92b34db-2297-4fc3-baab-7babcc0a94ff-kube-api-access-m5zrs\") pod \"dns-default-w4tml\" (UID: \"e92b34db-2297-4fc3-baab-7babcc0a94ff\") " pod="openshift-dns/dns-default-w4tml" Jul 13 22:14:00.251262 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:14:00.251071 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/2690217c-291d-4cbb-8ede-0ebe7802731c-ca-trust-extracted\") pod \"image-registry-74f88c6495-7kk5p\" (UID: \"2690217c-291d-4cbb-8ede-0ebe7802731c\") " pod="openshift-image-registry/image-registry-74f88c6495-7kk5p" Jul 13 22:14:00.251262 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:14:00.251090 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/e92b34db-2297-4fc3-baab-7babcc0a94ff-config-volume\") pod \"dns-default-w4tml\" (UID: \"e92b34db-2297-4fc3-baab-7babcc0a94ff\") " pod="openshift-dns/dns-default-w4tml" Jul 13 22:14:00.251262 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:14:00.251105 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/2690217c-291d-4cbb-8ede-0ebe7802731c-bound-sa-token\") pod \"image-registry-74f88c6495-7kk5p\" (UID: \"2690217c-291d-4cbb-8ede-0ebe7802731c\") " pod="openshift-image-registry/image-registry-74f88c6495-7kk5p" Jul 13 22:14:00.251262 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:14:00.251123 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/2690217c-291d-4cbb-8ede-0ebe7802731c-registry-certificates\") pod \"image-registry-74f88c6495-7kk5p\" (UID: \"2690217c-291d-4cbb-8ede-0ebe7802731c\") " pod="openshift-image-registry/image-registry-74f88c6495-7kk5p" Jul 13 22:14:00.351900 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:14:00.351810 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e92b34db-2297-4fc3-baab-7babcc0a94ff-tmp-dir\") pod \"dns-default-w4tml\" (UID: \"e92b34db-2297-4fc3-baab-7babcc0a94ff\") " pod="openshift-dns/dns-default-w4tml" Jul 13 22:14:00.351900 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:14:00.351855 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/2690217c-291d-4cbb-8ede-0ebe7802731c-installation-pull-secrets\") pod \"image-registry-74f88c6495-7kk5p\" (UID: \"2690217c-291d-4cbb-8ede-0ebe7802731c\") " pod="openshift-image-registry/image-registry-74f88c6495-7kk5p" Jul 13 22:14:00.352106 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:14:00.352059 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/e92b34db-2297-4fc3-baab-7babcc0a94ff-metrics-tls\") pod \"dns-default-w4tml\" (UID: \"e92b34db-2297-4fc3-baab-7babcc0a94ff\") " pod="openshift-dns/dns-default-w4tml" Jul 13 22:14:00.352158 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:14:00.352105 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/2690217c-291d-4cbb-8ede-0ebe7802731c-registry-tls\") pod \"image-registry-74f88c6495-7kk5p\" (UID: \"2690217c-291d-4cbb-8ede-0ebe7802731c\") " pod="openshift-image-registry/image-registry-74f88c6495-7kk5p" Jul 13 22:14:00.352225 ip-10-0-133-179 kubenswrapper[2583]: E0713 22:14:00.352202 2583 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 13 22:14:00.352225 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:14:00.352213 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e92b34db-2297-4fc3-baab-7babcc0a94ff-tmp-dir\") pod \"dns-default-w4tml\" (UID: \"e92b34db-2297-4fc3-baab-7babcc0a94ff\") " pod="openshift-dns/dns-default-w4tml" Jul 13 22:14:00.352225 ip-10-0-133-179 kubenswrapper[2583]: E0713 22:14:00.352224 2583 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-74f88c6495-7kk5p: secret "image-registry-tls" not found Jul 13 22:14:00.352439 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:14:00.352215 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/2690217c-291d-4cbb-8ede-0ebe7802731c-trusted-ca\") pod \"image-registry-74f88c6495-7kk5p\" (UID: \"2690217c-291d-4cbb-8ede-0ebe7802731c\") " pod="openshift-image-registry/image-registry-74f88c6495-7kk5p" Jul 13 22:14:00.352439 ip-10-0-133-179 kubenswrapper[2583]: E0713 22:14:00.352272 2583 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 13 22:14:00.352439 ip-10-0-133-179 kubenswrapper[2583]: E0713 22:14:00.352283 2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/2690217c-291d-4cbb-8ede-0ebe7802731c-registry-tls podName:2690217c-291d-4cbb-8ede-0ebe7802731c nodeName:}" failed. No retries permitted until 2026-07-13 22:14:00.85226241 +0000 UTC m=+36.100177752 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/2690217c-291d-4cbb-8ede-0ebe7802731c-registry-tls") pod "image-registry-74f88c6495-7kk5p" (UID: "2690217c-291d-4cbb-8ede-0ebe7802731c") : secret "image-registry-tls" not found Jul 13 22:14:00.352439 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:14:00.352302 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rjkqx\" (UniqueName: \"kubernetes.io/projected/f8f9da95-14d8-4f4f-8142-0a2f347ba336-kube-api-access-rjkqx\") pod \"ingress-canary-sbw7w\" (UID: \"f8f9da95-14d8-4f4f-8142-0a2f347ba336\") " pod="openshift-ingress-canary/ingress-canary-sbw7w" Jul 13 22:14:00.352439 ip-10-0-133-179 kubenswrapper[2583]: E0713 22:14:00.352332 2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e92b34db-2297-4fc3-baab-7babcc0a94ff-metrics-tls podName:e92b34db-2297-4fc3-baab-7babcc0a94ff nodeName:}" failed. No retries permitted until 2026-07-13 22:14:00.852310569 +0000 UTC m=+36.100225900 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/e92b34db-2297-4fc3-baab-7babcc0a94ff-metrics-tls") pod "dns-default-w4tml" (UID: "e92b34db-2297-4fc3-baab-7babcc0a94ff") : secret "dns-default-metrics-tls" not found Jul 13 22:14:00.352439 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:14:00.352374 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6lwmw\" (UniqueName: \"kubernetes.io/projected/2690217c-291d-4cbb-8ede-0ebe7802731c-kube-api-access-6lwmw\") pod \"image-registry-74f88c6495-7kk5p\" (UID: \"2690217c-291d-4cbb-8ede-0ebe7802731c\") " pod="openshift-image-registry/image-registry-74f88c6495-7kk5p" Jul 13 22:14:00.352439 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:14:00.352407 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/2690217c-291d-4cbb-8ede-0ebe7802731c-image-registry-private-configuration\") pod \"image-registry-74f88c6495-7kk5p\" (UID: \"2690217c-291d-4cbb-8ede-0ebe7802731c\") " pod="openshift-image-registry/image-registry-74f88c6495-7kk5p" Jul 13 22:14:00.352439 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:14:00.352435 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/f8f9da95-14d8-4f4f-8142-0a2f347ba336-cert\") pod \"ingress-canary-sbw7w\" (UID: \"f8f9da95-14d8-4f4f-8142-0a2f347ba336\") " pod="openshift-ingress-canary/ingress-canary-sbw7w" Jul 13 22:14:00.352822 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:14:00.352468 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m5zrs\" (UniqueName: \"kubernetes.io/projected/e92b34db-2297-4fc3-baab-7babcc0a94ff-kube-api-access-m5zrs\") pod \"dns-default-w4tml\" (UID: \"e92b34db-2297-4fc3-baab-7babcc0a94ff\") " pod="openshift-dns/dns-default-w4tml" Jul 13 22:14:00.352822 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:14:00.352495 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/2690217c-291d-4cbb-8ede-0ebe7802731c-ca-trust-extracted\") pod \"image-registry-74f88c6495-7kk5p\" (UID: \"2690217c-291d-4cbb-8ede-0ebe7802731c\") " pod="openshift-image-registry/image-registry-74f88c6495-7kk5p" Jul 13 22:14:00.352822 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:14:00.352525 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/e92b34db-2297-4fc3-baab-7babcc0a94ff-config-volume\") pod \"dns-default-w4tml\" (UID: \"e92b34db-2297-4fc3-baab-7babcc0a94ff\") " pod="openshift-dns/dns-default-w4tml" Jul 13 22:14:00.352822 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:14:00.352596 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/2690217c-291d-4cbb-8ede-0ebe7802731c-bound-sa-token\") pod \"image-registry-74f88c6495-7kk5p\" (UID: \"2690217c-291d-4cbb-8ede-0ebe7802731c\") " pod="openshift-image-registry/image-registry-74f88c6495-7kk5p" Jul 13 22:14:00.352822 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:14:00.352627 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/2690217c-291d-4cbb-8ede-0ebe7802731c-registry-certificates\") pod \"image-registry-74f88c6495-7kk5p\" (UID: \"2690217c-291d-4cbb-8ede-0ebe7802731c\") " pod="openshift-image-registry/image-registry-74f88c6495-7kk5p" Jul 13 22:14:00.353011 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:14:00.352873 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/2690217c-291d-4cbb-8ede-0ebe7802731c-ca-trust-extracted\") pod \"image-registry-74f88c6495-7kk5p\" (UID: \"2690217c-291d-4cbb-8ede-0ebe7802731c\") " pod="openshift-image-registry/image-registry-74f88c6495-7kk5p" Jul 13 22:14:00.353287 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:14:00.353192 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/2690217c-291d-4cbb-8ede-0ebe7802731c-trusted-ca\") pod \"image-registry-74f88c6495-7kk5p\" (UID: \"2690217c-291d-4cbb-8ede-0ebe7802731c\") " pod="openshift-image-registry/image-registry-74f88c6495-7kk5p" Jul 13 22:14:00.353287 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:14:00.353212 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/e92b34db-2297-4fc3-baab-7babcc0a94ff-config-volume\") pod \"dns-default-w4tml\" (UID: \"e92b34db-2297-4fc3-baab-7babcc0a94ff\") " pod="openshift-dns/dns-default-w4tml" Jul 13 22:14:00.353287 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:14:00.353200 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/2690217c-291d-4cbb-8ede-0ebe7802731c-registry-certificates\") pod \"image-registry-74f88c6495-7kk5p\" (UID: \"2690217c-291d-4cbb-8ede-0ebe7802731c\") " pod="openshift-image-registry/image-registry-74f88c6495-7kk5p" Jul 13 22:14:00.356729 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:14:00.356654 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/2690217c-291d-4cbb-8ede-0ebe7802731c-image-registry-private-configuration\") pod \"image-registry-74f88c6495-7kk5p\" (UID: \"2690217c-291d-4cbb-8ede-0ebe7802731c\") " pod="openshift-image-registry/image-registry-74f88c6495-7kk5p" Jul 13 22:14:00.356729 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:14:00.356678 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/f8f9da95-14d8-4f4f-8142-0a2f347ba336-cert\") pod \"ingress-canary-sbw7w\" (UID: \"f8f9da95-14d8-4f4f-8142-0a2f347ba336\") " pod="openshift-ingress-canary/ingress-canary-sbw7w" Jul 13 22:14:00.356729 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:14:00.356727 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/2690217c-291d-4cbb-8ede-0ebe7802731c-installation-pull-secrets\") pod \"image-registry-74f88c6495-7kk5p\" (UID: \"2690217c-291d-4cbb-8ede-0ebe7802731c\") " pod="openshift-image-registry/image-registry-74f88c6495-7kk5p" Jul 13 22:14:00.361109 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:14:00.361085 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/2690217c-291d-4cbb-8ede-0ebe7802731c-bound-sa-token\") pod \"image-registry-74f88c6495-7kk5p\" (UID: \"2690217c-291d-4cbb-8ede-0ebe7802731c\") " pod="openshift-image-registry/image-registry-74f88c6495-7kk5p" Jul 13 22:14:00.361267 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:14:00.361248 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-m5zrs\" (UniqueName: \"kubernetes.io/projected/e92b34db-2297-4fc3-baab-7babcc0a94ff-kube-api-access-m5zrs\") pod \"dns-default-w4tml\" (UID: \"e92b34db-2297-4fc3-baab-7babcc0a94ff\") " pod="openshift-dns/dns-default-w4tml" Jul 13 22:14:00.361765 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:14:00.361744 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6lwmw\" (UniqueName: \"kubernetes.io/projected/2690217c-291d-4cbb-8ede-0ebe7802731c-kube-api-access-6lwmw\") pod \"image-registry-74f88c6495-7kk5p\" (UID: \"2690217c-291d-4cbb-8ede-0ebe7802731c\") " pod="openshift-image-registry/image-registry-74f88c6495-7kk5p" Jul 13 22:14:00.361858 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:14:00.361741 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rjkqx\" (UniqueName: \"kubernetes.io/projected/f8f9da95-14d8-4f4f-8142-0a2f347ba336-kube-api-access-rjkqx\") pod \"ingress-canary-sbw7w\" (UID: \"f8f9da95-14d8-4f4f-8142-0a2f347ba336\") " pod="openshift-ingress-canary/ingress-canary-sbw7w" Jul 13 22:14:00.384678 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:14:00.384635 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-v8mkx" Jul 13 22:14:00.384851 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:14:00.384688 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-26w5h" Jul 13 22:14:00.384851 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:14:00.384635 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-9hl5q" Jul 13 22:14:00.387316 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:14:00.387292 2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Jul 13 22:14:00.387442 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:14:00.387292 2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Jul 13 22:14:00.387442 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:14:00.387411 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-cpxwn\"" Jul 13 22:14:00.387863 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:14:00.387829 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-jtd57\"" Jul 13 22:14:00.387970 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:14:00.387889 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Jul 13 22:14:00.387970 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:14:00.387921 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Jul 13 22:14:00.430238 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:14:00.430173 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-sbw7w" Jul 13 22:14:00.577940 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:14:00.577910 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-sbw7w"] Jul 13 22:14:00.581965 ip-10-0-133-179 kubenswrapper[2583]: W0713 22:14:00.581942 2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf8f9da95_14d8_4f4f_8142_0a2f347ba336.slice/crio-1c6d698530e3c5079afe4eb5b975f8298593bffe5ff106b1650586eb3397e098 WatchSource:0}: Error finding container 1c6d698530e3c5079afe4eb5b975f8298593bffe5ff106b1650586eb3397e098: Status 404 returned error can't find the container with id 1c6d698530e3c5079afe4eb5b975f8298593bffe5ff106b1650586eb3397e098 Jul 13 22:14:00.645190 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:14:00.645102 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-sbw7w" event={"ID":"f8f9da95-14d8-4f4f-8142-0a2f347ba336","Type":"ContainerStarted","Data":"1c6d698530e3c5079afe4eb5b975f8298593bffe5ff106b1650586eb3397e098"} Jul 13 22:14:00.856965 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:14:00.856927 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/e92b34db-2297-4fc3-baab-7babcc0a94ff-metrics-tls\") pod \"dns-default-w4tml\" (UID: \"e92b34db-2297-4fc3-baab-7babcc0a94ff\") " pod="openshift-dns/dns-default-w4tml" Jul 13 22:14:00.857142 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:14:00.856970 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/2690217c-291d-4cbb-8ede-0ebe7802731c-registry-tls\") pod \"image-registry-74f88c6495-7kk5p\" (UID: \"2690217c-291d-4cbb-8ede-0ebe7802731c\") " pod="openshift-image-registry/image-registry-74f88c6495-7kk5p" Jul 13 22:14:00.857142 ip-10-0-133-179 kubenswrapper[2583]: E0713 22:14:00.857100 2583 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 13 22:14:00.857254 ip-10-0-133-179 kubenswrapper[2583]: E0713 22:14:00.857164 2583 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 13 22:14:00.857254 ip-10-0-133-179 kubenswrapper[2583]: E0713 22:14:00.857175 2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e92b34db-2297-4fc3-baab-7babcc0a94ff-metrics-tls podName:e92b34db-2297-4fc3-baab-7babcc0a94ff nodeName:}" failed. No retries permitted until 2026-07-13 22:14:01.857153475 +0000 UTC m=+37.105068804 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/e92b34db-2297-4fc3-baab-7babcc0a94ff-metrics-tls") pod "dns-default-w4tml" (UID: "e92b34db-2297-4fc3-baab-7babcc0a94ff") : secret "dns-default-metrics-tls" not found Jul 13 22:14:00.857254 ip-10-0-133-179 kubenswrapper[2583]: E0713 22:14:00.857178 2583 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-74f88c6495-7kk5p: secret "image-registry-tls" not found Jul 13 22:14:00.857254 ip-10-0-133-179 kubenswrapper[2583]: E0713 22:14:00.857217 2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/2690217c-291d-4cbb-8ede-0ebe7802731c-registry-tls podName:2690217c-291d-4cbb-8ede-0ebe7802731c nodeName:}" failed. No retries permitted until 2026-07-13 22:14:01.857207358 +0000 UTC m=+37.105122689 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/2690217c-291d-4cbb-8ede-0ebe7802731c-registry-tls") pod "image-registry-74f88c6495-7kk5p" (UID: "2690217c-291d-4cbb-8ede-0ebe7802731c") : secret "image-registry-tls" not found Jul 13 22:14:01.108077 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:14:01.107849 2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator/migrator-7f59b54f6c-sfztp"] Jul 13 22:14:01.110303 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:14:01.110284 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-7f59b54f6c-sfztp" Jul 13 22:14:01.112627 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:14:01.112597 2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-root-ca.crt\"" Jul 13 22:14:01.113306 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:14:01.113277 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-storage-version-migrator-sa-dockercfg-q62j5\"" Jul 13 22:14:01.113306 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:14:01.113302 2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"openshift-service-ca.crt\"" Jul 13 22:14:01.119449 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:14:01.119427 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-7f59b54f6c-sfztp"] Jul 13 22:14:01.261046 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:14:01.261005 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j6jvk\" (UniqueName: \"kubernetes.io/projected/ad1974c1-a1f3-41bc-aed1-133b74495c7e-kube-api-access-j6jvk\") pod \"migrator-7f59b54f6c-sfztp\" (UID: \"ad1974c1-a1f3-41bc-aed1-133b74495c7e\") " pod="openshift-kube-storage-version-migrator/migrator-7f59b54f6c-sfztp" Jul 13 22:14:01.337581 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:14:01.337548 2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-dxppg"] Jul 13 22:14:01.339871 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:14:01.339843 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-dxppg" Jul 13 22:14:01.342058 ip-10-0-133-179 kubenswrapper[2583]: E0713 22:14:01.342023 2583 status_manager.go:1018] "Failed to get status for pod" err="pods \"insights-runtime-extractor-dxppg\" is forbidden: User \"system:node:ip-10-0-133-179.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"openshift-insights\": no relationship found between node 'ip-10-0-133-179.ec2.internal' and this object" podUID="479a52b0-c5da-471f-a153-02d4155abe62" pod="openshift-insights/insights-runtime-extractor-dxppg" Jul 13 22:14:01.342058 ip-10-0-133-179 kubenswrapper[2583]: E0713 22:14:01.342046 2583 reflector.go:205] "Failed to watch" err="failed to list *v1.Secret: secrets \"insights-runtime-extractor-tls\" is forbidden: User \"system:node:ip-10-0-133-179.ec2.internal\" cannot list resource \"secrets\" in API group \"\" in the namespace \"openshift-insights\": no relationship found between node 'ip-10-0-133-179.ec2.internal' and this object" logger="UnhandledError" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\"" type="*v1.Secret" Jul 13 22:14:01.342780 ip-10-0-133-179 kubenswrapper[2583]: E0713 22:14:01.342753 2583 reflector.go:205] "Failed to watch" err="failed to list *v1.ConfigMap: configmaps \"kube-root-ca.crt\" is forbidden: User \"system:node:ip-10-0-133-179.ec2.internal\" cannot list resource \"configmaps\" in API group \"\" in the namespace \"openshift-insights\": no relationship found between node 'ip-10-0-133-179.ec2.internal' and this object" logger="UnhandledError" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\"" type="*v1.ConfigMap" Jul 13 22:14:01.342877 ip-10-0-133-179 kubenswrapper[2583]: E0713 22:14:01.342780 2583 reflector.go:205] "Failed to watch" err="failed to list *v1.ConfigMap: configmaps \"kube-rbac-proxy\" is forbidden: User \"system:node:ip-10-0-133-179.ec2.internal\" cannot list resource \"configmaps\" in API group \"\" in the namespace \"openshift-insights\": no relationship found between node 'ip-10-0-133-179.ec2.internal' and this object" logger="UnhandledError" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\"" type="*v1.ConfigMap" Jul 13 22:14:01.343191 ip-10-0-133-179 kubenswrapper[2583]: E0713 22:14:01.343159 2583 reflector.go:205] "Failed to watch" err="failed to list *v1.Secret: secrets \"insights-runtime-extractor-sa-dockercfg-w5vtc\" is forbidden: User \"system:node:ip-10-0-133-179.ec2.internal\" cannot list resource \"secrets\" in API group \"\" in the namespace \"openshift-insights\": no relationship found between node 'ip-10-0-133-179.ec2.internal' and this object" logger="UnhandledError" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-w5vtc\"" type="*v1.Secret" Jul 13 22:14:01.343939 ip-10-0-133-179 kubenswrapper[2583]: E0713 22:14:01.343910 2583 reflector.go:205] "Failed to watch" err="failed to list *v1.ConfigMap: configmaps \"openshift-service-ca.crt\" is forbidden: User \"system:node:ip-10-0-133-179.ec2.internal\" cannot list resource \"configmaps\" in API group \"\" in the namespace \"openshift-insights\": no relationship found between node 'ip-10-0-133-179.ec2.internal' and this object" logger="UnhandledError" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\"" type="*v1.ConfigMap" Jul 13 22:14:01.361852 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:14:01.361755 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j6jvk\" (UniqueName: \"kubernetes.io/projected/ad1974c1-a1f3-41bc-aed1-133b74495c7e-kube-api-access-j6jvk\") pod \"migrator-7f59b54f6c-sfztp\" (UID: \"ad1974c1-a1f3-41bc-aed1-133b74495c7e\") " pod="openshift-kube-storage-version-migrator/migrator-7f59b54f6c-sfztp" Jul 13 22:14:01.369274 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:14:01.369237 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-dxppg"] Jul 13 22:14:01.375649 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:14:01.375622 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-j6jvk\" (UniqueName: \"kubernetes.io/projected/ad1974c1-a1f3-41bc-aed1-133b74495c7e-kube-api-access-j6jvk\") pod \"migrator-7f59b54f6c-sfztp\" (UID: \"ad1974c1-a1f3-41bc-aed1-133b74495c7e\") " pod="openshift-kube-storage-version-migrator/migrator-7f59b54f6c-sfztp" Jul 13 22:14:01.423495 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:14:01.423460 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-7f59b54f6c-sfztp" Jul 13 22:14:01.463115 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:14:01.463076 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/479a52b0-c5da-471f-a153-02d4155abe62-data-volume\") pod \"insights-runtime-extractor-dxppg\" (UID: \"479a52b0-c5da-471f-a153-02d4155abe62\") " pod="openshift-insights/insights-runtime-extractor-dxppg" Jul 13 22:14:01.463278 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:14:01.463121 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/479a52b0-c5da-471f-a153-02d4155abe62-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-dxppg\" (UID: \"479a52b0-c5da-471f-a153-02d4155abe62\") " pod="openshift-insights/insights-runtime-extractor-dxppg" Jul 13 22:14:01.463278 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:14:01.463194 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/479a52b0-c5da-471f-a153-02d4155abe62-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-dxppg\" (UID: \"479a52b0-c5da-471f-a153-02d4155abe62\") " pod="openshift-insights/insights-runtime-extractor-dxppg" Jul 13 22:14:01.463278 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:14:01.463255 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/479a52b0-c5da-471f-a153-02d4155abe62-crio-socket\") pod \"insights-runtime-extractor-dxppg\" (UID: \"479a52b0-c5da-471f-a153-02d4155abe62\") " pod="openshift-insights/insights-runtime-extractor-dxppg" Jul 13 22:14:01.463443 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:14:01.463298 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7x9qg\" (UniqueName: \"kubernetes.io/projected/479a52b0-c5da-471f-a153-02d4155abe62-kube-api-access-7x9qg\") pod \"insights-runtime-extractor-dxppg\" (UID: \"479a52b0-c5da-471f-a153-02d4155abe62\") " pod="openshift-insights/insights-runtime-extractor-dxppg" Jul 13 22:14:01.564024 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:14:01.563987 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/479a52b0-c5da-471f-a153-02d4155abe62-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-dxppg\" (UID: \"479a52b0-c5da-471f-a153-02d4155abe62\") " pod="openshift-insights/insights-runtime-extractor-dxppg" Jul 13 22:14:01.564182 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:14:01.564059 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/479a52b0-c5da-471f-a153-02d4155abe62-crio-socket\") pod \"insights-runtime-extractor-dxppg\" (UID: \"479a52b0-c5da-471f-a153-02d4155abe62\") " pod="openshift-insights/insights-runtime-extractor-dxppg" Jul 13 22:14:01.564182 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:14:01.564103 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7x9qg\" (UniqueName: \"kubernetes.io/projected/479a52b0-c5da-471f-a153-02d4155abe62-kube-api-access-7x9qg\") pod \"insights-runtime-extractor-dxppg\" (UID: \"479a52b0-c5da-471f-a153-02d4155abe62\") " pod="openshift-insights/insights-runtime-extractor-dxppg" Jul 13 22:14:01.564182 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:14:01.564172 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/479a52b0-c5da-471f-a153-02d4155abe62-data-volume\") pod \"insights-runtime-extractor-dxppg\" (UID: \"479a52b0-c5da-471f-a153-02d4155abe62\") " pod="openshift-insights/insights-runtime-extractor-dxppg" Jul 13 22:14:01.564322 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:14:01.564195 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/479a52b0-c5da-471f-a153-02d4155abe62-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-dxppg\" (UID: \"479a52b0-c5da-471f-a153-02d4155abe62\") " pod="openshift-insights/insights-runtime-extractor-dxppg" Jul 13 22:14:01.564361 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:14:01.564337 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/479a52b0-c5da-471f-a153-02d4155abe62-crio-socket\") pod \"insights-runtime-extractor-dxppg\" (UID: \"479a52b0-c5da-471f-a153-02d4155abe62\") " pod="openshift-insights/insights-runtime-extractor-dxppg" Jul 13 22:14:01.564631 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:14:01.564609 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/479a52b0-c5da-471f-a153-02d4155abe62-data-volume\") pod \"insights-runtime-extractor-dxppg\" (UID: \"479a52b0-c5da-471f-a153-02d4155abe62\") " pod="openshift-insights/insights-runtime-extractor-dxppg" Jul 13 22:14:01.867539 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:14:01.867505 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/e92b34db-2297-4fc3-baab-7babcc0a94ff-metrics-tls\") pod \"dns-default-w4tml\" (UID: \"e92b34db-2297-4fc3-baab-7babcc0a94ff\") " pod="openshift-dns/dns-default-w4tml" Jul 13 22:14:01.867539 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:14:01.867541 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/2690217c-291d-4cbb-8ede-0ebe7802731c-registry-tls\") pod \"image-registry-74f88c6495-7kk5p\" (UID: \"2690217c-291d-4cbb-8ede-0ebe7802731c\") " pod="openshift-image-registry/image-registry-74f88c6495-7kk5p" Jul 13 22:14:01.867796 ip-10-0-133-179 kubenswrapper[2583]: E0713 22:14:01.867679 2583 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 13 22:14:01.867796 ip-10-0-133-179 kubenswrapper[2583]: E0713 22:14:01.867699 2583 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 13 22:14:01.867796 ip-10-0-133-179 kubenswrapper[2583]: E0713 22:14:01.867712 2583 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-74f88c6495-7kk5p: secret "image-registry-tls" not found Jul 13 22:14:01.867796 ip-10-0-133-179 kubenswrapper[2583]: E0713 22:14:01.867750 2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e92b34db-2297-4fc3-baab-7babcc0a94ff-metrics-tls podName:e92b34db-2297-4fc3-baab-7babcc0a94ff nodeName:}" failed. No retries permitted until 2026-07-13 22:14:03.86773077 +0000 UTC m=+39.115646115 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/e92b34db-2297-4fc3-baab-7babcc0a94ff-metrics-tls") pod "dns-default-w4tml" (UID: "e92b34db-2297-4fc3-baab-7babcc0a94ff") : secret "dns-default-metrics-tls" not found Jul 13 22:14:01.867796 ip-10-0-133-179 kubenswrapper[2583]: E0713 22:14:01.867764 2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/2690217c-291d-4cbb-8ede-0ebe7802731c-registry-tls podName:2690217c-291d-4cbb-8ede-0ebe7802731c nodeName:}" failed. No retries permitted until 2026-07-13 22:14:03.867757807 +0000 UTC m=+39.115673135 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/2690217c-291d-4cbb-8ede-0ebe7802731c-registry-tls") pod "image-registry-74f88c6495-7kk5p" (UID: "2690217c-291d-4cbb-8ede-0ebe7802731c") : secret "image-registry-tls" not found Jul 13 22:14:02.209220 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:14:02.209144 2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\"" Jul 13 22:14:02.237798 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:14:02.237762 2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\"" Jul 13 22:14:02.244815 ip-10-0-133-179 kubenswrapper[2583]: E0713 22:14:02.244774 2583 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 13 22:14:02.244974 ip-10-0-133-179 kubenswrapper[2583]: E0713 22:14:02.244867 2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/479a52b0-c5da-471f-a153-02d4155abe62-kube-rbac-proxy-cm podName:479a52b0-c5da-471f-a153-02d4155abe62 nodeName:}" failed. No retries permitted until 2026-07-13 22:14:02.744844678 +0000 UTC m=+37.992760025 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/479a52b0-c5da-471f-a153-02d4155abe62-kube-rbac-proxy-cm") pod "insights-runtime-extractor-dxppg" (UID: "479a52b0-c5da-471f-a153-02d4155abe62") : configmap "kube-rbac-proxy" not found Jul 13 22:14:02.275999 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:14:02.275971 2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\"" Jul 13 22:14:02.288558 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:14:02.288526 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7x9qg\" (UniqueName: \"kubernetes.io/projected/479a52b0-c5da-471f-a153-02d4155abe62-kube-api-access-7x9qg\") pod \"insights-runtime-extractor-dxppg\" (UID: \"479a52b0-c5da-471f-a153-02d4155abe62\") " pod="openshift-insights/insights-runtime-extractor-dxppg" Jul 13 22:14:02.564743 ip-10-0-133-179 kubenswrapper[2583]: E0713 22:14:02.564702 2583 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: failed to sync secret cache: timed out waiting for the condition Jul 13 22:14:02.564922 ip-10-0-133-179 kubenswrapper[2583]: E0713 22:14:02.564808 2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/479a52b0-c5da-471f-a153-02d4155abe62-insights-runtime-extractor-tls podName:479a52b0-c5da-471f-a153-02d4155abe62 nodeName:}" failed. No retries permitted until 2026-07-13 22:14:03.064784775 +0000 UTC m=+38.312700129 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/479a52b0-c5da-471f-a153-02d4155abe62-insights-runtime-extractor-tls") pod "insights-runtime-extractor-dxppg" (UID: "479a52b0-c5da-471f-a153-02d4155abe62") : failed to sync secret cache: timed out waiting for the condition Jul 13 22:14:02.667459 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:14:02.667429 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-w5vtc\"" Jul 13 22:14:02.684535 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:14:02.684503 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\"" Jul 13 22:14:02.775403 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:14:02.775361 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/479a52b0-c5da-471f-a153-02d4155abe62-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-dxppg\" (UID: \"479a52b0-c5da-471f-a153-02d4155abe62\") " pod="openshift-insights/insights-runtime-extractor-dxppg" Jul 13 22:14:02.775655 ip-10-0-133-179 kubenswrapper[2583]: E0713 22:14:02.775497 2583 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 13 22:14:02.775655 ip-10-0-133-179 kubenswrapper[2583]: E0713 22:14:02.775602 2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/479a52b0-c5da-471f-a153-02d4155abe62-kube-rbac-proxy-cm podName:479a52b0-c5da-471f-a153-02d4155abe62 nodeName:}" failed. No retries permitted until 2026-07-13 22:14:03.775555358 +0000 UTC m=+39.023470703 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/479a52b0-c5da-471f-a153-02d4155abe62-kube-rbac-proxy-cm") pod "insights-runtime-extractor-dxppg" (UID: "479a52b0-c5da-471f-a153-02d4155abe62") : configmap "kube-rbac-proxy" not found Jul 13 22:14:03.079486 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:14:03.079443 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/479a52b0-c5da-471f-a153-02d4155abe62-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-dxppg\" (UID: \"479a52b0-c5da-471f-a153-02d4155abe62\") " pod="openshift-insights/insights-runtime-extractor-dxppg" Jul 13 22:14:03.079710 ip-10-0-133-179 kubenswrapper[2583]: E0713 22:14:03.079622 2583 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 13 22:14:03.079710 ip-10-0-133-179 kubenswrapper[2583]: E0713 22:14:03.079696 2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/479a52b0-c5da-471f-a153-02d4155abe62-insights-runtime-extractor-tls podName:479a52b0-c5da-471f-a153-02d4155abe62 nodeName:}" failed. No retries permitted until 2026-07-13 22:14:04.079676295 +0000 UTC m=+39.327591623 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/479a52b0-c5da-471f-a153-02d4155abe62-insights-runtime-extractor-tls") pod "insights-runtime-extractor-dxppg" (UID: "479a52b0-c5da-471f-a153-02d4155abe62") : secret "insights-runtime-extractor-tls" not found Jul 13 22:14:03.290509 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:14:03.290471 2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-service-ca/service-ca-79f68b9f4f-hhrnn"] Jul 13 22:14:03.303736 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:14:03.303704 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-79f68b9f4f-hhrnn" Jul 13 22:14:03.310990 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:14:03.310962 2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"signing-cabundle\"" Jul 13 22:14:03.311214 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:14:03.311187 2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"kube-root-ca.crt\"" Jul 13 22:14:03.312111 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:14:03.312048 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"signing-key\"" Jul 13 22:14:03.312111 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:14:03.312065 2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"openshift-service-ca.crt\"" Jul 13 22:14:03.312111 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:14:03.312070 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"service-ca-dockercfg-dw9th\"" Jul 13 22:14:03.318928 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:14:03.318901 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-79f68b9f4f-hhrnn"] Jul 13 22:14:03.483005 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:14:03.482920 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/7d9f697c-99ac-4c4d-944f-69ea038ced1f-signing-cabundle\") pod \"service-ca-79f68b9f4f-hhrnn\" (UID: \"7d9f697c-99ac-4c4d-944f-69ea038ced1f\") " pod="openshift-service-ca/service-ca-79f68b9f4f-hhrnn" Jul 13 22:14:03.483005 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:14:03.482959 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lh24q\" (UniqueName: \"kubernetes.io/projected/7d9f697c-99ac-4c4d-944f-69ea038ced1f-kube-api-access-lh24q\") pod \"service-ca-79f68b9f4f-hhrnn\" (UID: \"7d9f697c-99ac-4c4d-944f-69ea038ced1f\") " pod="openshift-service-ca/service-ca-79f68b9f4f-hhrnn" Jul 13 22:14:03.483185 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:14:03.483013 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/7d9f697c-99ac-4c4d-944f-69ea038ced1f-signing-key\") pod \"service-ca-79f68b9f4f-hhrnn\" (UID: \"7d9f697c-99ac-4c4d-944f-69ea038ced1f\") " pod="openshift-service-ca/service-ca-79f68b9f4f-hhrnn" Jul 13 22:14:03.584435 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:14:03.584273 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/7d9f697c-99ac-4c4d-944f-69ea038ced1f-signing-cabundle\") pod \"service-ca-79f68b9f4f-hhrnn\" (UID: \"7d9f697c-99ac-4c4d-944f-69ea038ced1f\") " pod="openshift-service-ca/service-ca-79f68b9f4f-hhrnn" Jul 13 22:14:03.584435 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:14:03.584307 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lh24q\" (UniqueName: \"kubernetes.io/projected/7d9f697c-99ac-4c4d-944f-69ea038ced1f-kube-api-access-lh24q\") pod \"service-ca-79f68b9f4f-hhrnn\" (UID: \"7d9f697c-99ac-4c4d-944f-69ea038ced1f\") " pod="openshift-service-ca/service-ca-79f68b9f4f-hhrnn" Jul 13 22:14:03.584435 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:14:03.584363 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/7d9f697c-99ac-4c4d-944f-69ea038ced1f-signing-key\") pod \"service-ca-79f68b9f4f-hhrnn\" (UID: \"7d9f697c-99ac-4c4d-944f-69ea038ced1f\") " pod="openshift-service-ca/service-ca-79f68b9f4f-hhrnn" Jul 13 22:14:03.584947 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:14:03.584924 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/7d9f697c-99ac-4c4d-944f-69ea038ced1f-signing-cabundle\") pod \"service-ca-79f68b9f4f-hhrnn\" (UID: \"7d9f697c-99ac-4c4d-944f-69ea038ced1f\") " pod="openshift-service-ca/service-ca-79f68b9f4f-hhrnn" Jul 13 22:14:03.586916 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:14:03.586891 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/7d9f697c-99ac-4c4d-944f-69ea038ced1f-signing-key\") pod \"service-ca-79f68b9f4f-hhrnn\" (UID: \"7d9f697c-99ac-4c4d-944f-69ea038ced1f\") " pod="openshift-service-ca/service-ca-79f68b9f4f-hhrnn" Jul 13 22:14:03.594350 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:14:03.594329 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lh24q\" (UniqueName: \"kubernetes.io/projected/7d9f697c-99ac-4c4d-944f-69ea038ced1f-kube-api-access-lh24q\") pod \"service-ca-79f68b9f4f-hhrnn\" (UID: \"7d9f697c-99ac-4c4d-944f-69ea038ced1f\") " pod="openshift-service-ca/service-ca-79f68b9f4f-hhrnn" Jul 13 22:14:03.615412 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:14:03.615380 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-79f68b9f4f-hhrnn" Jul 13 22:14:03.775561 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:14:03.775528 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-7f59b54f6c-sfztp"] Jul 13 22:14:03.786199 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:14:03.786161 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/479a52b0-c5da-471f-a153-02d4155abe62-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-dxppg\" (UID: \"479a52b0-c5da-471f-a153-02d4155abe62\") " pod="openshift-insights/insights-runtime-extractor-dxppg" Jul 13 22:14:03.786383 ip-10-0-133-179 kubenswrapper[2583]: E0713 22:14:03.786270 2583 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 13 22:14:03.786383 ip-10-0-133-179 kubenswrapper[2583]: E0713 22:14:03.786333 2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/479a52b0-c5da-471f-a153-02d4155abe62-kube-rbac-proxy-cm podName:479a52b0-c5da-471f-a153-02d4155abe62 nodeName:}" failed. No retries permitted until 2026-07-13 22:14:05.786314301 +0000 UTC m=+41.034229635 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/479a52b0-c5da-471f-a153-02d4155abe62-kube-rbac-proxy-cm") pod "insights-runtime-extractor-dxppg" (UID: "479a52b0-c5da-471f-a153-02d4155abe62") : configmap "kube-rbac-proxy" not found Jul 13 22:14:03.816682 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:14:03.816562 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-79f68b9f4f-hhrnn"] Jul 13 22:14:03.821758 ip-10-0-133-179 kubenswrapper[2583]: W0713 22:14:03.821722 2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podad1974c1_a1f3_41bc_aed1_133b74495c7e.slice/crio-c413dba1c14b3695bdf1146274be0a1915df8ecbedb2accb551f01a89d461d01 WatchSource:0}: Error finding container c413dba1c14b3695bdf1146274be0a1915df8ecbedb2accb551f01a89d461d01: Status 404 returned error can't find the container with id c413dba1c14b3695bdf1146274be0a1915df8ecbedb2accb551f01a89d461d01 Jul 13 22:14:03.822189 ip-10-0-133-179 kubenswrapper[2583]: W0713 22:14:03.822169 2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7d9f697c_99ac_4c4d_944f_69ea038ced1f.slice/crio-8e97699f20f45747beaf12e21ca396e77bd2bfec175e950da88c1e3c78df05cb WatchSource:0}: Error finding container 8e97699f20f45747beaf12e21ca396e77bd2bfec175e950da88c1e3c78df05cb: Status 404 returned error can't find the container with id 8e97699f20f45747beaf12e21ca396e77bd2bfec175e950da88c1e3c78df05cb Jul 13 22:14:03.886775 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:14:03.886738 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/e92b34db-2297-4fc3-baab-7babcc0a94ff-metrics-tls\") pod \"dns-default-w4tml\" (UID: \"e92b34db-2297-4fc3-baab-7babcc0a94ff\") " pod="openshift-dns/dns-default-w4tml" Jul 13 22:14:03.886775 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:14:03.886788 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/2690217c-291d-4cbb-8ede-0ebe7802731c-registry-tls\") pod \"image-registry-74f88c6495-7kk5p\" (UID: \"2690217c-291d-4cbb-8ede-0ebe7802731c\") " pod="openshift-image-registry/image-registry-74f88c6495-7kk5p" Jul 13 22:14:03.886972 ip-10-0-133-179 kubenswrapper[2583]: E0713 22:14:03.886923 2583 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 13 22:14:03.886972 ip-10-0-133-179 kubenswrapper[2583]: E0713 22:14:03.886948 2583 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 13 22:14:03.886972 ip-10-0-133-179 kubenswrapper[2583]: E0713 22:14:03.886965 2583 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-74f88c6495-7kk5p: secret "image-registry-tls" not found Jul 13 22:14:03.887086 ip-10-0-133-179 kubenswrapper[2583]: E0713 22:14:03.887001 2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e92b34db-2297-4fc3-baab-7babcc0a94ff-metrics-tls podName:e92b34db-2297-4fc3-baab-7babcc0a94ff nodeName:}" failed. No retries permitted until 2026-07-13 22:14:07.88697972 +0000 UTC m=+43.134895048 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/e92b34db-2297-4fc3-baab-7babcc0a94ff-metrics-tls") pod "dns-default-w4tml" (UID: "e92b34db-2297-4fc3-baab-7babcc0a94ff") : secret "dns-default-metrics-tls" not found Jul 13 22:14:03.887086 ip-10-0-133-179 kubenswrapper[2583]: E0713 22:14:03.887020 2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/2690217c-291d-4cbb-8ede-0ebe7802731c-registry-tls podName:2690217c-291d-4cbb-8ede-0ebe7802731c nodeName:}" failed. No retries permitted until 2026-07-13 22:14:07.887010351 +0000 UTC m=+43.134925686 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/2690217c-291d-4cbb-8ede-0ebe7802731c-registry-tls") pod "image-registry-74f88c6495-7kk5p" (UID: "2690217c-291d-4cbb-8ede-0ebe7802731c") : secret "image-registry-tls" not found Jul 13 22:14:03.909848 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:14:03.909726 2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-grjf8_e8e90a01-fd99-4a55-ba17-522966f4fabe/dns-node-resolver/0.log" Jul 13 22:14:04.088399 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:14:04.088368 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/479a52b0-c5da-471f-a153-02d4155abe62-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-dxppg\" (UID: \"479a52b0-c5da-471f-a153-02d4155abe62\") " pod="openshift-insights/insights-runtime-extractor-dxppg" Jul 13 22:14:04.088585 ip-10-0-133-179 kubenswrapper[2583]: E0713 22:14:04.088525 2583 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 13 22:14:04.088630 ip-10-0-133-179 kubenswrapper[2583]: E0713 22:14:04.088616 2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/479a52b0-c5da-471f-a153-02d4155abe62-insights-runtime-extractor-tls podName:479a52b0-c5da-471f-a153-02d4155abe62 nodeName:}" failed. No retries permitted until 2026-07-13 22:14:06.088600238 +0000 UTC m=+41.336515566 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/479a52b0-c5da-471f-a153-02d4155abe62-insights-runtime-extractor-tls") pod "insights-runtime-extractor-dxppg" (UID: "479a52b0-c5da-471f-a153-02d4155abe62") : secret "insights-runtime-extractor-tls" not found Jul 13 22:14:04.669533 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:14:04.669430 2583 generic.go:356] "Generic (PLEG): container finished" podID="03f58cdb-4d8b-41a9-aa82-dff50f158f0c" containerID="4fca5b381a8873749e3ac678dadbb06aa4c3d4d5ec43da1ee8e7e6a4bff0b555" exitCode=0 Jul 13 22:14:04.669533 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:14:04.669514 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-nbdsh" event={"ID":"03f58cdb-4d8b-41a9-aa82-dff50f158f0c","Type":"ContainerDied","Data":"4fca5b381a8873749e3ac678dadbb06aa4c3d4d5ec43da1ee8e7e6a4bff0b555"} Jul 13 22:14:04.671878 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:14:04.671850 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-sbw7w" event={"ID":"f8f9da95-14d8-4f4f-8142-0a2f347ba336","Type":"ContainerStarted","Data":"405aff6a934ef77a2a17fb3a90bdc94c66e0c22e75a81e022131ef0581caddc3"} Jul 13 22:14:04.673599 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:14:04.673553 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-7f59b54f6c-sfztp" event={"ID":"ad1974c1-a1f3-41bc-aed1-133b74495c7e","Type":"ContainerStarted","Data":"c413dba1c14b3695bdf1146274be0a1915df8ecbedb2accb551f01a89d461d01"} Jul 13 22:14:04.675013 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:14:04.674985 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-79f68b9f4f-hhrnn" event={"ID":"7d9f697c-99ac-4c4d-944f-69ea038ced1f","Type":"ContainerStarted","Data":"8e97699f20f45747beaf12e21ca396e77bd2bfec175e950da88c1e3c78df05cb"} Jul 13 22:14:04.716617 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:14:04.716540 2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-sbw7w" podStartSLOduration=1.69709577 podStartE2EDuration="4.716518314s" totalImagesPullingTime="3.019422544s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-13 22:14:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 22:14:04.715446162 +0000 UTC m=+39.963361542" watchObservedRunningTime="2026-07-13 22:14:04.716518314 +0000 UTC m=+39.964433665" Jul 13 22:14:04.908009 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:14:04.907940 2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-r26w6_54e8db97-3d72-4aa1-80bb-e56f10c7e474/node-ca/0.log" Jul 13 22:14:05.510191 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:14:05.510151 2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-sbw7w_f8f9da95-14d8-4f4f-8142-0a2f347ba336/serve-healthcheck-canary/0.log" Jul 13 22:14:05.680989 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:14:05.680952 2583 generic.go:356] "Generic (PLEG): container finished" podID="03f58cdb-4d8b-41a9-aa82-dff50f158f0c" containerID="73ace3b927c44c8290fc5fec8f1484eb6d283c15ba4fd0d1fa5d974d7da14afe" exitCode=0 Jul 13 22:14:05.681619 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:14:05.681040 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-nbdsh" event={"ID":"03f58cdb-4d8b-41a9-aa82-dff50f158f0c","Type":"ContainerDied","Data":"73ace3b927c44c8290fc5fec8f1484eb6d283c15ba4fd0d1fa5d974d7da14afe"} Jul 13 22:14:05.802502 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:14:05.802454 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/479a52b0-c5da-471f-a153-02d4155abe62-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-dxppg\" (UID: \"479a52b0-c5da-471f-a153-02d4155abe62\") " pod="openshift-insights/insights-runtime-extractor-dxppg" Jul 13 22:14:05.802700 ip-10-0-133-179 kubenswrapper[2583]: E0713 22:14:05.802635 2583 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 13 22:14:05.802758 ip-10-0-133-179 kubenswrapper[2583]: E0713 22:14:05.802715 2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/479a52b0-c5da-471f-a153-02d4155abe62-kube-rbac-proxy-cm podName:479a52b0-c5da-471f-a153-02d4155abe62 nodeName:}" failed. No retries permitted until 2026-07-13 22:14:09.802693426 +0000 UTC m=+45.050608762 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/479a52b0-c5da-471f-a153-02d4155abe62-kube-rbac-proxy-cm") pod "insights-runtime-extractor-dxppg" (UID: "479a52b0-c5da-471f-a153-02d4155abe62") : configmap "kube-rbac-proxy" not found Jul 13 22:14:06.104334 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:14:06.104309 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/479a52b0-c5da-471f-a153-02d4155abe62-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-dxppg\" (UID: \"479a52b0-c5da-471f-a153-02d4155abe62\") " pod="openshift-insights/insights-runtime-extractor-dxppg" Jul 13 22:14:06.104436 ip-10-0-133-179 kubenswrapper[2583]: E0713 22:14:06.104425 2583 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 13 22:14:06.104482 ip-10-0-133-179 kubenswrapper[2583]: E0713 22:14:06.104474 2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/479a52b0-c5da-471f-a153-02d4155abe62-insights-runtime-extractor-tls podName:479a52b0-c5da-471f-a153-02d4155abe62 nodeName:}" failed. No retries permitted until 2026-07-13 22:14:10.104460849 +0000 UTC m=+45.352376177 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/479a52b0-c5da-471f-a153-02d4155abe62-insights-runtime-extractor-tls") pod "insights-runtime-extractor-dxppg" (UID: "479a52b0-c5da-471f-a153-02d4155abe62") : secret "insights-runtime-extractor-tls" not found Jul 13 22:14:06.686812 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:14:06.686714 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-nbdsh" event={"ID":"03f58cdb-4d8b-41a9-aa82-dff50f158f0c","Type":"ContainerStarted","Data":"b5f1bc8d082bf59eb6e0b029fe62a8b8d67a42ab2b5d8f77998ac38a130e9cce"} Jul 13 22:14:06.688517 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:14:06.688489 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-7f59b54f6c-sfztp" event={"ID":"ad1974c1-a1f3-41bc-aed1-133b74495c7e","Type":"ContainerStarted","Data":"3c1ef837b528f3e2c78926bc228ffde7cc240108aa5d51e0b4eca68c1457ff1d"} Jul 13 22:14:06.688663 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:14:06.688550 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-7f59b54f6c-sfztp" event={"ID":"ad1974c1-a1f3-41bc-aed1-133b74495c7e","Type":"ContainerStarted","Data":"14b1049b27e18d8d7b24ef3cb6456a4de85d682992918ec651ffeb73bb77a12a"} Jul 13 22:14:06.689850 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:14:06.689824 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-79f68b9f4f-hhrnn" event={"ID":"7d9f697c-99ac-4c4d-944f-69ea038ced1f","Type":"ContainerStarted","Data":"9f181728903b13f58d9f69613c2b897abfa89ba43228163e2a2b9f9c706cb2e6"} Jul 13 22:14:06.709382 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:14:06.709330 2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-nbdsh" podStartSLOduration=9.971516128 podStartE2EDuration="41.709311598s" totalImagesPullingTime="31.73779547s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-13 22:13:25 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 22:14:06.70883895 +0000 UTC m=+41.956754298" watchObservedRunningTime="2026-07-13 22:14:06.709311598 +0000 UTC m=+41.957226948" Jul 13 22:14:06.724935 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:14:06.724889 2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator/migrator-7f59b54f6c-sfztp" podStartSLOduration=3.455752595 podStartE2EDuration="5.724874734s" totalImagesPullingTime="2.269122139s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-13 22:14:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 22:14:06.723917621 +0000 UTC m=+41.971832999" watchObservedRunningTime="2026-07-13 22:14:06.724874734 +0000 UTC m=+41.972790084" Jul 13 22:14:06.743504 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:14:06.743457 2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-service-ca/service-ca-79f68b9f4f-hhrnn" podStartSLOduration=1.211314037 podStartE2EDuration="3.743440416s" totalImagesPullingTime="2.532126379s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-13 22:14:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 22:14:06.742534845 +0000 UTC m=+41.990450205" watchObservedRunningTime="2026-07-13 22:14:06.743440416 +0000 UTC m=+41.991355744" Jul 13 22:14:07.921093 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:14:07.921056 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/e92b34db-2297-4fc3-baab-7babcc0a94ff-metrics-tls\") pod \"dns-default-w4tml\" (UID: \"e92b34db-2297-4fc3-baab-7babcc0a94ff\") " pod="openshift-dns/dns-default-w4tml" Jul 13 22:14:07.921093 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:14:07.921103 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/2690217c-291d-4cbb-8ede-0ebe7802731c-registry-tls\") pod \"image-registry-74f88c6495-7kk5p\" (UID: \"2690217c-291d-4cbb-8ede-0ebe7802731c\") " pod="openshift-image-registry/image-registry-74f88c6495-7kk5p" Jul 13 22:14:07.921623 ip-10-0-133-179 kubenswrapper[2583]: E0713 22:14:07.921232 2583 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 13 22:14:07.921623 ip-10-0-133-179 kubenswrapper[2583]: E0713 22:14:07.921296 2583 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 13 22:14:07.921623 ip-10-0-133-179 kubenswrapper[2583]: E0713 22:14:07.921309 2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e92b34db-2297-4fc3-baab-7babcc0a94ff-metrics-tls podName:e92b34db-2297-4fc3-baab-7babcc0a94ff nodeName:}" failed. No retries permitted until 2026-07-13 22:14:15.92128843 +0000 UTC m=+51.169203770 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/e92b34db-2297-4fc3-baab-7babcc0a94ff-metrics-tls") pod "dns-default-w4tml" (UID: "e92b34db-2297-4fc3-baab-7babcc0a94ff") : secret "dns-default-metrics-tls" not found Jul 13 22:14:07.921623 ip-10-0-133-179 kubenswrapper[2583]: E0713 22:14:07.921312 2583 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-74f88c6495-7kk5p: secret "image-registry-tls" not found Jul 13 22:14:07.921623 ip-10-0-133-179 kubenswrapper[2583]: E0713 22:14:07.921351 2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/2690217c-291d-4cbb-8ede-0ebe7802731c-registry-tls podName:2690217c-291d-4cbb-8ede-0ebe7802731c nodeName:}" failed. No retries permitted until 2026-07-13 22:14:15.921341373 +0000 UTC m=+51.169256704 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/2690217c-291d-4cbb-8ede-0ebe7802731c-registry-tls") pod "image-registry-74f88c6495-7kk5p" (UID: "2690217c-291d-4cbb-8ede-0ebe7802731c") : secret "image-registry-tls" not found Jul 13 22:14:09.838401 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:14:09.838361 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/479a52b0-c5da-471f-a153-02d4155abe62-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-dxppg\" (UID: \"479a52b0-c5da-471f-a153-02d4155abe62\") " pod="openshift-insights/insights-runtime-extractor-dxppg" Jul 13 22:14:09.839058 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:14:09.839031 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/479a52b0-c5da-471f-a153-02d4155abe62-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-dxppg\" (UID: \"479a52b0-c5da-471f-a153-02d4155abe62\") " pod="openshift-insights/insights-runtime-extractor-dxppg" Jul 13 22:14:10.141964 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:14:10.141872 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/479a52b0-c5da-471f-a153-02d4155abe62-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-dxppg\" (UID: \"479a52b0-c5da-471f-a153-02d4155abe62\") " pod="openshift-insights/insights-runtime-extractor-dxppg" Jul 13 22:14:10.142112 ip-10-0-133-179 kubenswrapper[2583]: E0713 22:14:10.142023 2583 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 13 22:14:10.142112 ip-10-0-133-179 kubenswrapper[2583]: E0713 22:14:10.142086 2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/479a52b0-c5da-471f-a153-02d4155abe62-insights-runtime-extractor-tls podName:479a52b0-c5da-471f-a153-02d4155abe62 nodeName:}" failed. No retries permitted until 2026-07-13 22:14:18.142069965 +0000 UTC m=+53.389985312 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/479a52b0-c5da-471f-a153-02d4155abe62-insights-runtime-extractor-tls") pod "insights-runtime-extractor-dxppg" (UID: "479a52b0-c5da-471f-a153-02d4155abe62") : secret "insights-runtime-extractor-tls" not found Jul 13 22:14:15.986958 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:14:15.986921 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/e92b34db-2297-4fc3-baab-7babcc0a94ff-metrics-tls\") pod \"dns-default-w4tml\" (UID: \"e92b34db-2297-4fc3-baab-7babcc0a94ff\") " pod="openshift-dns/dns-default-w4tml" Jul 13 22:14:15.986958 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:14:15.986960 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/2690217c-291d-4cbb-8ede-0ebe7802731c-registry-tls\") pod \"image-registry-74f88c6495-7kk5p\" (UID: \"2690217c-291d-4cbb-8ede-0ebe7802731c\") " pod="openshift-image-registry/image-registry-74f88c6495-7kk5p" Jul 13 22:14:15.990129 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:14:15.990099 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/e92b34db-2297-4fc3-baab-7babcc0a94ff-metrics-tls\") pod \"dns-default-w4tml\" (UID: \"e92b34db-2297-4fc3-baab-7babcc0a94ff\") " pod="openshift-dns/dns-default-w4tml" Jul 13 22:14:15.990300 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:14:15.990151 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/2690217c-291d-4cbb-8ede-0ebe7802731c-registry-tls\") pod \"image-registry-74f88c6495-7kk5p\" (UID: \"2690217c-291d-4cbb-8ede-0ebe7802731c\") " pod="openshift-image-registry/image-registry-74f88c6495-7kk5p" Jul 13 22:14:16.015499 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:14:16.015459 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-74f88c6495-7kk5p" Jul 13 22:14:16.023647 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:14:16.023626 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-w4tml" Jul 13 22:14:16.203301 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:14:16.203252 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-w4tml"] Jul 13 22:14:16.207388 ip-10-0-133-179 kubenswrapper[2583]: W0713 22:14:16.207361 2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode92b34db_2297_4fc3_baab_7babcc0a94ff.slice/crio-bfd951475645d726bc97204665c404e8e1a090e517544c92f42a50db5fc0d6fd WatchSource:0}: Error finding container bfd951475645d726bc97204665c404e8e1a090e517544c92f42a50db5fc0d6fd: Status 404 returned error can't find the container with id bfd951475645d726bc97204665c404e8e1a090e517544c92f42a50db5fc0d6fd Jul 13 22:14:16.213381 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:14:16.213354 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-74f88c6495-7kk5p"] Jul 13 22:14:16.216384 ip-10-0-133-179 kubenswrapper[2583]: W0713 22:14:16.216363 2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2690217c_291d_4cbb_8ede_0ebe7802731c.slice/crio-9fd137bfdb99741f9a2b4744b86a2e7fa43f9e5689dc30ee1bad10f9a55910c9 WatchSource:0}: Error finding container 9fd137bfdb99741f9a2b4744b86a2e7fa43f9e5689dc30ee1bad10f9a55910c9: Status 404 returned error can't find the container with id 9fd137bfdb99741f9a2b4744b86a2e7fa43f9e5689dc30ee1bad10f9a55910c9 Jul 13 22:14:16.715640 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:14:16.715609 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-w4tml" event={"ID":"e92b34db-2297-4fc3-baab-7babcc0a94ff","Type":"ContainerStarted","Data":"bfd951475645d726bc97204665c404e8e1a090e517544c92f42a50db5fc0d6fd"} Jul 13 22:14:16.716893 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:14:16.716868 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-74f88c6495-7kk5p" event={"ID":"2690217c-291d-4cbb-8ede-0ebe7802731c","Type":"ContainerStarted","Data":"406329a37adb98218fcda890e8ea3fd13d421dcce93ca7eb3db39be5fc1c58bd"} Jul 13 22:14:16.717006 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:14:16.716907 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-74f88c6495-7kk5p" event={"ID":"2690217c-291d-4cbb-8ede-0ebe7802731c","Type":"ContainerStarted","Data":"9fd137bfdb99741f9a2b4744b86a2e7fa43f9e5689dc30ee1bad10f9a55910c9"} Jul 13 22:14:16.717051 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:14:16.717006 2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-74f88c6495-7kk5p" Jul 13 22:14:16.738981 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:14:16.738922 2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/image-registry-74f88c6495-7kk5p" podStartSLOduration=32.738900777 podStartE2EDuration="32.738900777s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-13 22:13:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 22:14:16.73848943 +0000 UTC m=+51.986404781" watchObservedRunningTime="2026-07-13 22:14:16.738900777 +0000 UTC m=+51.986816163" Jul 13 22:14:18.206165 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:14:18.206123 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/479a52b0-c5da-471f-a153-02d4155abe62-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-dxppg\" (UID: \"479a52b0-c5da-471f-a153-02d4155abe62\") " pod="openshift-insights/insights-runtime-extractor-dxppg" Jul 13 22:14:18.210037 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:14:18.209503 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/479a52b0-c5da-471f-a153-02d4155abe62-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-dxppg\" (UID: \"479a52b0-c5da-471f-a153-02d4155abe62\") " pod="openshift-insights/insights-runtime-extractor-dxppg" Jul 13 22:14:18.451326 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:14:18.451286 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-dxppg" Jul 13 22:14:18.876793 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:14:18.876757 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-dxppg"] Jul 13 22:14:18.880823 ip-10-0-133-179 kubenswrapper[2583]: W0713 22:14:18.880786 2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod479a52b0_c5da_471f_a153_02d4155abe62.slice/crio-62f5dac1bf1ac1246bb4bbb22c58142013d7c4e42f28f8b7afa4ec0f50de11f6 WatchSource:0}: Error finding container 62f5dac1bf1ac1246bb4bbb22c58142013d7c4e42f28f8b7afa4ec0f50de11f6: Status 404 returned error can't find the container with id 62f5dac1bf1ac1246bb4bbb22c58142013d7c4e42f28f8b7afa4ec0f50de11f6 Jul 13 22:14:19.726077 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:14:19.726041 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-dxppg" event={"ID":"479a52b0-c5da-471f-a153-02d4155abe62","Type":"ContainerStarted","Data":"62f5dac1bf1ac1246bb4bbb22c58142013d7c4e42f28f8b7afa4ec0f50de11f6"} Jul 13 22:14:19.727745 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:14:19.727723 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-w4tml" event={"ID":"e92b34db-2297-4fc3-baab-7babcc0a94ff","Type":"ContainerStarted","Data":"ee486bcedf238bc5957a191c9c96350db1ad1216c40df771781eeffc556e5f69"} Jul 13 22:14:19.727846 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:14:19.727767 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-w4tml" event={"ID":"e92b34db-2297-4fc3-baab-7babcc0a94ff","Type":"ContainerStarted","Data":"4e24b7afce6551a0eeba464ac1426e8ca57d2991da8b5a0efcacd493cfac6dce"} Jul 13 22:14:19.727846 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:14:19.727801 2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-w4tml" Jul 13 22:14:19.750361 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:14:19.750305 2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-w4tml" podStartSLOduration=16.906737954 podStartE2EDuration="19.750290519s" totalImagesPullingTime="2.843552565s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-13 22:14:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 22:14:19.748992809 +0000 UTC m=+54.996908170" watchObservedRunningTime="2026-07-13 22:14:19.750290519 +0000 UTC m=+54.998205869" Jul 13 22:14:24.744321 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:14:24.744284 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-dxppg" event={"ID":"479a52b0-c5da-471f-a153-02d4155abe62","Type":"ContainerStarted","Data":"fe7b0ff0c390b392551ee120d32adfc6822e53e52008086c2fd1c4464b2aed6e"} Jul 13 22:14:25.376392 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:14:25.376349 2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5c447f47c5-kdt2l"] Jul 13 22:14:25.397793 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:14:25.397764 2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-55ff8454b8-xf6vg"] Jul 13 22:14:25.397948 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:14:25.397887 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5c447f47c5-kdt2l" Jul 13 22:14:25.402957 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:14:25.402923 2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"kube-root-ca.crt\"" Jul 13 22:14:25.403143 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:14:25.403100 2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"openshift-service-ca.crt\"" Jul 13 22:14:25.403143 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:14:25.403111 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-service-proxy-server-certificates\"" Jul 13 22:14:25.403251 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:14:25.403216 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-hub-kubeconfig\"" Jul 13 22:14:25.403365 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:14:25.403350 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-ca\"" Jul 13 22:14:25.403449 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:14:25.403431 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"open-cluster-management-image-pull-credentials\"" Jul 13 22:14:25.403625 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:14:25.403608 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-open-cluster-management.io-proxy-agent-signer-client-cert\"" Jul 13 22:14:25.413267 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:14:25.413243 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-55ff8454b8-xf6vg"] Jul 13 22:14:25.413408 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:14:25.413276 2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-74f88c6495-7kk5p"] Jul 13 22:14:25.413408 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:14:25.413351 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-55ff8454b8-xf6vg" Jul 13 22:14:25.415903 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:14:25.415870 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"managed-serviceaccount-hub-kubeconfig\"" Jul 13 22:14:25.416001 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:14:25.415870 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"managed-serviceaccount-dockercfg-gqqhs\"" Jul 13 22:14:25.417164 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:14:25.417147 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5c447f47c5-kdt2l"] Jul 13 22:14:25.463593 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:14:25.463538 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6gzkq\" (UniqueName: \"kubernetes.io/projected/b0c6afbd-639c-41d9-839d-a02ee7881ebf-kube-api-access-6gzkq\") pod \"managed-serviceaccount-addon-agent-55ff8454b8-xf6vg\" (UID: \"b0c6afbd-639c-41d9-839d-a02ee7881ebf\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-55ff8454b8-xf6vg" Jul 13 22:14:25.463779 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:14:25.463633 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/9ae4a320-42bc-4aef-a11e-d670ae0f8da7-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-5c447f47c5-kdt2l\" (UID: \"9ae4a320-42bc-4aef-a11e-d670ae0f8da7\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5c447f47c5-kdt2l" Jul 13 22:14:25.463779 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:14:25.463665 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/9ae4a320-42bc-4aef-a11e-d670ae0f8da7-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-5c447f47c5-kdt2l\" (UID: \"9ae4a320-42bc-4aef-a11e-d670ae0f8da7\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5c447f47c5-kdt2l" Jul 13 22:14:25.463779 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:14:25.463688 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/9ae4a320-42bc-4aef-a11e-d670ae0f8da7-additional-service-ca\") pod \"cluster-proxy-proxy-agent-5c447f47c5-kdt2l\" (UID: \"9ae4a320-42bc-4aef-a11e-d670ae0f8da7\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5c447f47c5-kdt2l" Jul 13 22:14:25.463779 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:14:25.463715 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/b0c6afbd-639c-41d9-839d-a02ee7881ebf-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-55ff8454b8-xf6vg\" (UID: \"b0c6afbd-639c-41d9-839d-a02ee7881ebf\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-55ff8454b8-xf6vg" Jul 13 22:14:25.463922 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:14:25.463775 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sbsg5\" (UniqueName: \"kubernetes.io/projected/9ae4a320-42bc-4aef-a11e-d670ae0f8da7-kube-api-access-sbsg5\") pod \"cluster-proxy-proxy-agent-5c447f47c5-kdt2l\" (UID: \"9ae4a320-42bc-4aef-a11e-d670ae0f8da7\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5c447f47c5-kdt2l" Jul 13 22:14:25.463922 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:14:25.463814 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca\" (UniqueName: \"kubernetes.io/secret/9ae4a320-42bc-4aef-a11e-d670ae0f8da7-ca\") pod \"cluster-proxy-proxy-agent-5c447f47c5-kdt2l\" (UID: \"9ae4a320-42bc-4aef-a11e-d670ae0f8da7\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5c447f47c5-kdt2l" Jul 13 22:14:25.463922 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:14:25.463879 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub\" (UniqueName: \"kubernetes.io/secret/9ae4a320-42bc-4aef-a11e-d670ae0f8da7-hub\") pod \"cluster-proxy-proxy-agent-5c447f47c5-kdt2l\" (UID: \"9ae4a320-42bc-4aef-a11e-d670ae0f8da7\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5c447f47c5-kdt2l" Jul 13 22:14:25.480824 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:14:25.480784 2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-f7c956786-wvmw5"] Jul 13 22:14:25.503871 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:14:25.503787 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-f7c956786-wvmw5"] Jul 13 22:14:25.504015 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:14:25.503894 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-f7c956786-wvmw5" Jul 13 22:14:25.565164 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:14:25.565131 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub\" (UniqueName: \"kubernetes.io/secret/9ae4a320-42bc-4aef-a11e-d670ae0f8da7-hub\") pod \"cluster-proxy-proxy-agent-5c447f47c5-kdt2l\" (UID: \"9ae4a320-42bc-4aef-a11e-d670ae0f8da7\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5c447f47c5-kdt2l" Jul 13 22:14:25.565164 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:14:25.565168 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/0ebe20b4-d6b9-404a-ac6f-a5c7894148cc-ca-trust-extracted\") pod \"image-registry-f7c956786-wvmw5\" (UID: \"0ebe20b4-d6b9-404a-ac6f-a5c7894148cc\") " pod="openshift-image-registry/image-registry-f7c956786-wvmw5" Jul 13 22:14:25.565367 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:14:25.565189 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6gzkq\" (UniqueName: \"kubernetes.io/projected/b0c6afbd-639c-41d9-839d-a02ee7881ebf-kube-api-access-6gzkq\") pod \"managed-serviceaccount-addon-agent-55ff8454b8-xf6vg\" (UID: \"b0c6afbd-639c-41d9-839d-a02ee7881ebf\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-55ff8454b8-xf6vg" Jul 13 22:14:25.565367 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:14:25.565211 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/0ebe20b4-d6b9-404a-ac6f-a5c7894148cc-bound-sa-token\") pod \"image-registry-f7c956786-wvmw5\" (UID: \"0ebe20b4-d6b9-404a-ac6f-a5c7894148cc\") " pod="openshift-image-registry/image-registry-f7c956786-wvmw5" Jul 13 22:14:25.565367 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:14:25.565247 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/9ae4a320-42bc-4aef-a11e-d670ae0f8da7-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-5c447f47c5-kdt2l\" (UID: \"9ae4a320-42bc-4aef-a11e-d670ae0f8da7\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5c447f47c5-kdt2l" Jul 13 22:14:25.565367 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:14:25.565264 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/0ebe20b4-d6b9-404a-ac6f-a5c7894148cc-registry-certificates\") pod \"image-registry-f7c956786-wvmw5\" (UID: \"0ebe20b4-d6b9-404a-ac6f-a5c7894148cc\") " pod="openshift-image-registry/image-registry-f7c956786-wvmw5" Jul 13 22:14:25.565367 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:14:25.565293 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/9ae4a320-42bc-4aef-a11e-d670ae0f8da7-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-5c447f47c5-kdt2l\" (UID: \"9ae4a320-42bc-4aef-a11e-d670ae0f8da7\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5c447f47c5-kdt2l" Jul 13 22:14:25.565367 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:14:25.565316 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/9ae4a320-42bc-4aef-a11e-d670ae0f8da7-additional-service-ca\") pod \"cluster-proxy-proxy-agent-5c447f47c5-kdt2l\" (UID: \"9ae4a320-42bc-4aef-a11e-d670ae0f8da7\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5c447f47c5-kdt2l" Jul 13 22:14:25.565697 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:14:25.565380 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/b0c6afbd-639c-41d9-839d-a02ee7881ebf-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-55ff8454b8-xf6vg\" (UID: \"b0c6afbd-639c-41d9-839d-a02ee7881ebf\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-55ff8454b8-xf6vg" Jul 13 22:14:25.565697 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:14:25.565420 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sbsg5\" (UniqueName: \"kubernetes.io/projected/9ae4a320-42bc-4aef-a11e-d670ae0f8da7-kube-api-access-sbsg5\") pod \"cluster-proxy-proxy-agent-5c447f47c5-kdt2l\" (UID: \"9ae4a320-42bc-4aef-a11e-d670ae0f8da7\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5c447f47c5-kdt2l" Jul 13 22:14:25.565697 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:14:25.565462 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/0ebe20b4-d6b9-404a-ac6f-a5c7894148cc-image-registry-private-configuration\") pod \"image-registry-f7c956786-wvmw5\" (UID: \"0ebe20b4-d6b9-404a-ac6f-a5c7894148cc\") " pod="openshift-image-registry/image-registry-f7c956786-wvmw5" Jul 13 22:14:25.565697 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:14:25.565490 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/0ebe20b4-d6b9-404a-ac6f-a5c7894148cc-trusted-ca\") pod \"image-registry-f7c956786-wvmw5\" (UID: \"0ebe20b4-d6b9-404a-ac6f-a5c7894148cc\") " pod="openshift-image-registry/image-registry-f7c956786-wvmw5" Jul 13 22:14:25.565697 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:14:25.565520 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cb62k\" (UniqueName: \"kubernetes.io/projected/0ebe20b4-d6b9-404a-ac6f-a5c7894148cc-kube-api-access-cb62k\") pod \"image-registry-f7c956786-wvmw5\" (UID: \"0ebe20b4-d6b9-404a-ac6f-a5c7894148cc\") " pod="openshift-image-registry/image-registry-f7c956786-wvmw5" Jul 13 22:14:25.565697 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:14:25.565551 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca\" (UniqueName: \"kubernetes.io/secret/9ae4a320-42bc-4aef-a11e-d670ae0f8da7-ca\") pod \"cluster-proxy-proxy-agent-5c447f47c5-kdt2l\" (UID: \"9ae4a320-42bc-4aef-a11e-d670ae0f8da7\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5c447f47c5-kdt2l" Jul 13 22:14:25.565697 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:14:25.565626 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/0ebe20b4-d6b9-404a-ac6f-a5c7894148cc-registry-tls\") pod \"image-registry-f7c956786-wvmw5\" (UID: \"0ebe20b4-d6b9-404a-ac6f-a5c7894148cc\") " pod="openshift-image-registry/image-registry-f7c956786-wvmw5" Jul 13 22:14:25.565697 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:14:25.565665 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/0ebe20b4-d6b9-404a-ac6f-a5c7894148cc-installation-pull-secrets\") pod \"image-registry-f7c956786-wvmw5\" (UID: \"0ebe20b4-d6b9-404a-ac6f-a5c7894148cc\") " pod="openshift-image-registry/image-registry-f7c956786-wvmw5" Jul 13 22:14:25.566121 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:14:25.566099 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/9ae4a320-42bc-4aef-a11e-d670ae0f8da7-additional-service-ca\") pod \"cluster-proxy-proxy-agent-5c447f47c5-kdt2l\" (UID: \"9ae4a320-42bc-4aef-a11e-d670ae0f8da7\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5c447f47c5-kdt2l" Jul 13 22:14:25.567875 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:14:25.567853 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/9ae4a320-42bc-4aef-a11e-d670ae0f8da7-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-5c447f47c5-kdt2l\" (UID: \"9ae4a320-42bc-4aef-a11e-d670ae0f8da7\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5c447f47c5-kdt2l" Jul 13 22:14:25.568094 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:14:25.568077 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca\" (UniqueName: \"kubernetes.io/secret/9ae4a320-42bc-4aef-a11e-d670ae0f8da7-ca\") pod \"cluster-proxy-proxy-agent-5c447f47c5-kdt2l\" (UID: \"9ae4a320-42bc-4aef-a11e-d670ae0f8da7\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5c447f47c5-kdt2l" Jul 13 22:14:25.568271 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:14:25.568245 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/9ae4a320-42bc-4aef-a11e-d670ae0f8da7-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-5c447f47c5-kdt2l\" (UID: \"9ae4a320-42bc-4aef-a11e-d670ae0f8da7\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5c447f47c5-kdt2l" Jul 13 22:14:25.568346 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:14:25.568301 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/b0c6afbd-639c-41d9-839d-a02ee7881ebf-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-55ff8454b8-xf6vg\" (UID: \"b0c6afbd-639c-41d9-839d-a02ee7881ebf\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-55ff8454b8-xf6vg" Jul 13 22:14:25.568346 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:14:25.568317 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hub\" (UniqueName: \"kubernetes.io/secret/9ae4a320-42bc-4aef-a11e-d670ae0f8da7-hub\") pod \"cluster-proxy-proxy-agent-5c447f47c5-kdt2l\" (UID: \"9ae4a320-42bc-4aef-a11e-d670ae0f8da7\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5c447f47c5-kdt2l" Jul 13 22:14:25.588487 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:14:25.588454 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6gzkq\" (UniqueName: \"kubernetes.io/projected/b0c6afbd-639c-41d9-839d-a02ee7881ebf-kube-api-access-6gzkq\") pod \"managed-serviceaccount-addon-agent-55ff8454b8-xf6vg\" (UID: \"b0c6afbd-639c-41d9-839d-a02ee7881ebf\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-55ff8454b8-xf6vg" Jul 13 22:14:25.591953 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:14:25.591932 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sbsg5\" (UniqueName: \"kubernetes.io/projected/9ae4a320-42bc-4aef-a11e-d670ae0f8da7-kube-api-access-sbsg5\") pod \"cluster-proxy-proxy-agent-5c447f47c5-kdt2l\" (UID: \"9ae4a320-42bc-4aef-a11e-d670ae0f8da7\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5c447f47c5-kdt2l" Jul 13 22:14:25.643074 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:14:25.643049 2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-khllb" Jul 13 22:14:25.666882 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:14:25.666850 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/0ebe20b4-d6b9-404a-ac6f-a5c7894148cc-bound-sa-token\") pod \"image-registry-f7c956786-wvmw5\" (UID: \"0ebe20b4-d6b9-404a-ac6f-a5c7894148cc\") " pod="openshift-image-registry/image-registry-f7c956786-wvmw5" Jul 13 22:14:25.667057 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:14:25.666917 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/0ebe20b4-d6b9-404a-ac6f-a5c7894148cc-registry-certificates\") pod \"image-registry-f7c956786-wvmw5\" (UID: \"0ebe20b4-d6b9-404a-ac6f-a5c7894148cc\") " pod="openshift-image-registry/image-registry-f7c956786-wvmw5" Jul 13 22:14:25.667057 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:14:25.666999 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/0ebe20b4-d6b9-404a-ac6f-a5c7894148cc-image-registry-private-configuration\") pod \"image-registry-f7c956786-wvmw5\" (UID: \"0ebe20b4-d6b9-404a-ac6f-a5c7894148cc\") " pod="openshift-image-registry/image-registry-f7c956786-wvmw5" Jul 13 22:14:25.667422 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:14:25.667390 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/0ebe20b4-d6b9-404a-ac6f-a5c7894148cc-trusted-ca\") pod \"image-registry-f7c956786-wvmw5\" (UID: \"0ebe20b4-d6b9-404a-ac6f-a5c7894148cc\") " pod="openshift-image-registry/image-registry-f7c956786-wvmw5" Jul 13 22:14:25.667532 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:14:25.667427 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cb62k\" (UniqueName: \"kubernetes.io/projected/0ebe20b4-d6b9-404a-ac6f-a5c7894148cc-kube-api-access-cb62k\") pod \"image-registry-f7c956786-wvmw5\" (UID: \"0ebe20b4-d6b9-404a-ac6f-a5c7894148cc\") " pod="openshift-image-registry/image-registry-f7c956786-wvmw5" Jul 13 22:14:25.667532 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:14:25.667465 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/0ebe20b4-d6b9-404a-ac6f-a5c7894148cc-registry-tls\") pod \"image-registry-f7c956786-wvmw5\" (UID: \"0ebe20b4-d6b9-404a-ac6f-a5c7894148cc\") " pod="openshift-image-registry/image-registry-f7c956786-wvmw5" Jul 13 22:14:25.667532 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:14:25.667490 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/0ebe20b4-d6b9-404a-ac6f-a5c7894148cc-installation-pull-secrets\") pod \"image-registry-f7c956786-wvmw5\" (UID: \"0ebe20b4-d6b9-404a-ac6f-a5c7894148cc\") " pod="openshift-image-registry/image-registry-f7c956786-wvmw5" Jul 13 22:14:25.667738 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:14:25.667600 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/0ebe20b4-d6b9-404a-ac6f-a5c7894148cc-ca-trust-extracted\") pod \"image-registry-f7c956786-wvmw5\" (UID: \"0ebe20b4-d6b9-404a-ac6f-a5c7894148cc\") " pod="openshift-image-registry/image-registry-f7c956786-wvmw5" Jul 13 22:14:25.667945 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:14:25.667923 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/0ebe20b4-d6b9-404a-ac6f-a5c7894148cc-ca-trust-extracted\") pod \"image-registry-f7c956786-wvmw5\" (UID: \"0ebe20b4-d6b9-404a-ac6f-a5c7894148cc\") " pod="openshift-image-registry/image-registry-f7c956786-wvmw5" Jul 13 22:14:25.668666 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:14:25.668640 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/0ebe20b4-d6b9-404a-ac6f-a5c7894148cc-registry-certificates\") pod \"image-registry-f7c956786-wvmw5\" (UID: \"0ebe20b4-d6b9-404a-ac6f-a5c7894148cc\") " pod="openshift-image-registry/image-registry-f7c956786-wvmw5" Jul 13 22:14:25.668927 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:14:25.668907 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/0ebe20b4-d6b9-404a-ac6f-a5c7894148cc-trusted-ca\") pod \"image-registry-f7c956786-wvmw5\" (UID: \"0ebe20b4-d6b9-404a-ac6f-a5c7894148cc\") " pod="openshift-image-registry/image-registry-f7c956786-wvmw5" Jul 13 22:14:25.669978 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:14:25.669948 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/0ebe20b4-d6b9-404a-ac6f-a5c7894148cc-image-registry-private-configuration\") pod \"image-registry-f7c956786-wvmw5\" (UID: \"0ebe20b4-d6b9-404a-ac6f-a5c7894148cc\") " pod="openshift-image-registry/image-registry-f7c956786-wvmw5" Jul 13 22:14:25.670218 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:14:25.670188 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/0ebe20b4-d6b9-404a-ac6f-a5c7894148cc-installation-pull-secrets\") pod \"image-registry-f7c956786-wvmw5\" (UID: \"0ebe20b4-d6b9-404a-ac6f-a5c7894148cc\") " pod="openshift-image-registry/image-registry-f7c956786-wvmw5" Jul 13 22:14:25.670319 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:14:25.670303 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/0ebe20b4-d6b9-404a-ac6f-a5c7894148cc-registry-tls\") pod \"image-registry-f7c956786-wvmw5\" (UID: \"0ebe20b4-d6b9-404a-ac6f-a5c7894148cc\") " pod="openshift-image-registry/image-registry-f7c956786-wvmw5" Jul 13 22:14:25.685860 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:14:25.685842 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cb62k\" (UniqueName: \"kubernetes.io/projected/0ebe20b4-d6b9-404a-ac6f-a5c7894148cc-kube-api-access-cb62k\") pod \"image-registry-f7c956786-wvmw5\" (UID: \"0ebe20b4-d6b9-404a-ac6f-a5c7894148cc\") " pod="openshift-image-registry/image-registry-f7c956786-wvmw5" Jul 13 22:14:25.685943 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:14:25.685880 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/0ebe20b4-d6b9-404a-ac6f-a5c7894148cc-bound-sa-token\") pod \"image-registry-f7c956786-wvmw5\" (UID: \"0ebe20b4-d6b9-404a-ac6f-a5c7894148cc\") " pod="openshift-image-registry/image-registry-f7c956786-wvmw5" Jul 13 22:14:25.725551 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:14:25.725518 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5c447f47c5-kdt2l" Jul 13 22:14:25.732672 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:14:25.732646 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-55ff8454b8-xf6vg" Jul 13 22:14:25.748660 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:14:25.748634 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-dxppg" event={"ID":"479a52b0-c5da-471f-a153-02d4155abe62","Type":"ContainerStarted","Data":"ad078a85daeb58fe22eaa71750ac74eeda8610b4f76512e9de83a7c721aa6463"} Jul 13 22:14:25.824028 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:14:25.823997 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-f7c956786-wvmw5" Jul 13 22:14:25.877464 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:14:25.877432 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5c447f47c5-kdt2l"] Jul 13 22:14:25.881047 ip-10-0-133-179 kubenswrapper[2583]: W0713 22:14:25.881011 2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod9ae4a320_42bc_4aef_a11e_d670ae0f8da7.slice/crio-b016afea17d55adb47b4d07177b9a7bdb99e63288af079037a3b90ec9c8a545b WatchSource:0}: Error finding container b016afea17d55adb47b4d07177b9a7bdb99e63288af079037a3b90ec9c8a545b: Status 404 returned error can't find the container with id b016afea17d55adb47b4d07177b9a7bdb99e63288af079037a3b90ec9c8a545b Jul 13 22:14:25.893528 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:14:25.893499 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-55ff8454b8-xf6vg"] Jul 13 22:14:25.897453 ip-10-0-133-179 kubenswrapper[2583]: W0713 22:14:25.897421 2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podb0c6afbd_639c_41d9_839d_a02ee7881ebf.slice/crio-e1896d3d5a83a92e7b9b034254ddad3f7164bf9f9edf6be00ad3a441b793f904 WatchSource:0}: Error finding container e1896d3d5a83a92e7b9b034254ddad3f7164bf9f9edf6be00ad3a441b793f904: Status 404 returned error can't find the container with id e1896d3d5a83a92e7b9b034254ddad3f7164bf9f9edf6be00ad3a441b793f904 Jul 13 22:14:25.986515 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:14:25.986483 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-f7c956786-wvmw5"] Jul 13 22:14:25.992100 ip-10-0-133-179 kubenswrapper[2583]: W0713 22:14:25.992070 2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0ebe20b4_d6b9_404a_ac6f_a5c7894148cc.slice/crio-2decf98a6ca9cd03290998c33dc843146361710d37835cbcb42b91a458fd6827 WatchSource:0}: Error finding container 2decf98a6ca9cd03290998c33dc843146361710d37835cbcb42b91a458fd6827: Status 404 returned error can't find the container with id 2decf98a6ca9cd03290998c33dc843146361710d37835cbcb42b91a458fd6827 Jul 13 22:14:26.753354 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:14:26.753311 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-55ff8454b8-xf6vg" event={"ID":"b0c6afbd-639c-41d9-839d-a02ee7881ebf","Type":"ContainerStarted","Data":"e1896d3d5a83a92e7b9b034254ddad3f7164bf9f9edf6be00ad3a441b793f904"} Jul 13 22:14:26.754751 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:14:26.754718 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5c447f47c5-kdt2l" event={"ID":"9ae4a320-42bc-4aef-a11e-d670ae0f8da7","Type":"ContainerStarted","Data":"b016afea17d55adb47b4d07177b9a7bdb99e63288af079037a3b90ec9c8a545b"} Jul 13 22:14:26.756479 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:14:26.756454 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-f7c956786-wvmw5" event={"ID":"0ebe20b4-d6b9-404a-ac6f-a5c7894148cc","Type":"ContainerStarted","Data":"bd454678f7dc2adbf5fd4d6d8a62727f67bcea1e29e249ecb2facf25e9d7a05f"} Jul 13 22:14:26.756756 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:14:26.756538 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-f7c956786-wvmw5" event={"ID":"0ebe20b4-d6b9-404a-ac6f-a5c7894148cc","Type":"ContainerStarted","Data":"2decf98a6ca9cd03290998c33dc843146361710d37835cbcb42b91a458fd6827"} Jul 13 22:14:26.756756 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:14:26.756629 2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-f7c956786-wvmw5" Jul 13 22:14:26.778403 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:14:26.778341 2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/image-registry-f7c956786-wvmw5" podStartSLOduration=1.778324113 podStartE2EDuration="1.778324113s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-13 22:14:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 22:14:26.777058889 +0000 UTC m=+62.024974241" watchObservedRunningTime="2026-07-13 22:14:26.778324113 +0000 UTC m=+62.026239467" Jul 13 22:14:29.733843 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:14:29.733816 2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-w4tml" Jul 13 22:14:30.777231 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:14:30.777196 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-55ff8454b8-xf6vg" event={"ID":"b0c6afbd-639c-41d9-839d-a02ee7881ebf","Type":"ContainerStarted","Data":"1269eb1e0f3cf3ff763b54cb605766008d78c4ab8f45f4f8b2b056184db143ee"} Jul 13 22:14:30.779980 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:14:30.779953 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5c447f47c5-kdt2l" event={"ID":"9ae4a320-42bc-4aef-a11e-d670ae0f8da7","Type":"ContainerStarted","Data":"e9a6f88f849f93e9f2e8cd624036fd397d7f80d1b759b05ec078dcd9a149c8c5"} Jul 13 22:14:30.780124 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:14:30.779993 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5c447f47c5-kdt2l" event={"ID":"9ae4a320-42bc-4aef-a11e-d670ae0f8da7","Type":"ContainerStarted","Data":"99d01c05ccd10e104a56878e7b8d0ac4147ab9f03df6e3b68e5c9b97fd59a0c5"} Jul 13 22:14:30.780124 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:14:30.780015 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5c447f47c5-kdt2l" event={"ID":"9ae4a320-42bc-4aef-a11e-d670ae0f8da7","Type":"ContainerStarted","Data":"4151051595d8f748b188cf61ef8e77354f279a1ba8fb3694d01d623a7bc8f525"} Jul 13 22:14:30.781959 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:14:30.781933 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-dxppg" event={"ID":"479a52b0-c5da-471f-a153-02d4155abe62","Type":"ContainerStarted","Data":"3311fb297be86ad46267044c8854617f0ef77a9e56714c30cfccc9e5dd898139"} Jul 13 22:14:30.792899 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:14:30.792855 2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-55ff8454b8-xf6vg" podStartSLOduration=1.132799755 podStartE2EDuration="5.792840543s" totalImagesPullingTime="4.660040788s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-13 22:14:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 22:14:30.792548571 +0000 UTC m=+66.040463922" watchObservedRunningTime="2026-07-13 22:14:30.792840543 +0000 UTC m=+66.040755871" Jul 13 22:14:30.817318 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:14:30.817212 2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5c447f47c5-kdt2l" podStartSLOduration=1.125473822 podStartE2EDuration="5.817193157s" totalImagesPullingTime="4.691719335s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-13 22:14:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 22:14:30.81684436 +0000 UTC m=+66.064759711" watchObservedRunningTime="2026-07-13 22:14:30.817193157 +0000 UTC m=+66.065108510" Jul 13 22:14:30.836263 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:14:30.836211 2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-dxppg" podStartSLOduration=18.338143344 podStartE2EDuration="29.836194941s" totalImagesPullingTime="11.498051597s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-13 22:14:01 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 22:14:30.835630938 +0000 UTC m=+66.083546307" watchObservedRunningTime="2026-07-13 22:14:30.836194941 +0000 UTC m=+66.084110283" Jul 13 22:14:31.016875 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:14:31.016836 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/d38c850d-d65b-4f99-ad24-5c1d443bff4b-original-pull-secret\") pod \"global-pull-secret-syncer-9hl5q\" (UID: \"d38c850d-d65b-4f99-ad24-5c1d443bff4b\") " pod="kube-system/global-pull-secret-syncer-9hl5q" Jul 13 22:14:31.019156 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:14:31.019129 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Jul 13 22:14:31.030194 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:14:31.030165 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/d38c850d-d65b-4f99-ad24-5c1d443bff4b-original-pull-secret\") pod \"global-pull-secret-syncer-9hl5q\" (UID: \"d38c850d-d65b-4f99-ad24-5c1d443bff4b\") " pod="kube-system/global-pull-secret-syncer-9hl5q" Jul 13 22:14:31.118326 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:14:31.118225 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tcrhd\" (UniqueName: \"kubernetes.io/projected/c8bbc3a9-6b5c-485d-b7ef-ca4cdca462ce-kube-api-access-tcrhd\") pod \"network-check-target-v8mkx\" (UID: \"c8bbc3a9-6b5c-485d-b7ef-ca4cdca462ce\") " pod="openshift-network-diagnostics/network-check-target-v8mkx" Jul 13 22:14:31.118326 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:14:31.118269 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/85265522-e114-4c04-b595-a26fcae86884-metrics-certs\") pod \"network-metrics-daemon-26w5h\" (UID: \"85265522-e114-4c04-b595-a26fcae86884\") " pod="openshift-multus/network-metrics-daemon-26w5h" Jul 13 22:14:31.120303 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:14:31.120279 2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Jul 13 22:14:31.120431 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:14:31.120346 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Jul 13 22:14:31.131114 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:14:31.131090 2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Jul 13 22:14:31.131202 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:14:31.131145 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/85265522-e114-4c04-b595-a26fcae86884-metrics-certs\") pod \"network-metrics-daemon-26w5h\" (UID: \"85265522-e114-4c04-b595-a26fcae86884\") " pod="openshift-multus/network-metrics-daemon-26w5h" Jul 13 22:14:31.141785 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:14:31.141757 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tcrhd\" (UniqueName: \"kubernetes.io/projected/c8bbc3a9-6b5c-485d-b7ef-ca4cdca462ce-kube-api-access-tcrhd\") pod \"network-check-target-v8mkx\" (UID: \"c8bbc3a9-6b5c-485d-b7ef-ca4cdca462ce\") " pod="openshift-network-diagnostics/network-check-target-v8mkx" Jul 13 22:14:31.299449 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:14:31.299414 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-cpxwn\"" Jul 13 22:14:31.307382 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:14:31.307348 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-v8mkx" Jul 13 22:14:31.307499 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:14:31.307385 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-9hl5q" Jul 13 22:14:31.318599 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:14:31.318562 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-jtd57\"" Jul 13 22:14:31.323631 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:14:31.323599 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-26w5h" Jul 13 22:14:31.461665 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:14:31.461562 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-9hl5q"] Jul 13 22:14:31.499421 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:14:31.499391 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-v8mkx"] Jul 13 22:14:31.502978 ip-10-0-133-179 kubenswrapper[2583]: W0713 22:14:31.502950 2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc8bbc3a9_6b5c_485d_b7ef_ca4cdca462ce.slice/crio-cac103293adf8d40182703b3260209fb9edef9b69cc119aa0b5cfe15e05465c9 WatchSource:0}: Error finding container cac103293adf8d40182703b3260209fb9edef9b69cc119aa0b5cfe15e05465c9: Status 404 returned error can't find the container with id cac103293adf8d40182703b3260209fb9edef9b69cc119aa0b5cfe15e05465c9 Jul 13 22:14:31.515449 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:14:31.515425 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-26w5h"] Jul 13 22:14:31.519455 ip-10-0-133-179 kubenswrapper[2583]: W0713 22:14:31.519433 2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod85265522_e114_4c04_b595_a26fcae86884.slice/crio-8404d57e1735eb8f56234ca92670f02cc6bc514c03f3729da516621e1467f1ca WatchSource:0}: Error finding container 8404d57e1735eb8f56234ca92670f02cc6bc514c03f3729da516621e1467f1ca: Status 404 returned error can't find the container with id 8404d57e1735eb8f56234ca92670f02cc6bc514c03f3729da516621e1467f1ca Jul 13 22:14:31.786564 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:14:31.786526 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-9hl5q" event={"ID":"d38c850d-d65b-4f99-ad24-5c1d443bff4b","Type":"ContainerStarted","Data":"7bb363cbd664e1f1e245a8926f95bd0368c60bc5e6aa943242f3e125c905d465"} Jul 13 22:14:31.787580 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:14:31.787541 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-26w5h" event={"ID":"85265522-e114-4c04-b595-a26fcae86884","Type":"ContainerStarted","Data":"8404d57e1735eb8f56234ca92670f02cc6bc514c03f3729da516621e1467f1ca"} Jul 13 22:14:31.788760 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:14:31.788737 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-v8mkx" event={"ID":"c8bbc3a9-6b5c-485d-b7ef-ca4cdca462ce","Type":"ContainerStarted","Data":"cac103293adf8d40182703b3260209fb9edef9b69cc119aa0b5cfe15e05465c9"} Jul 13 22:14:33.802979 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:14:33.802939 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-26w5h" event={"ID":"85265522-e114-4c04-b595-a26fcae86884","Type":"ContainerStarted","Data":"6e26715954ace748315f9120daece613df154625cf68c3f10d2da1a0c9d97e9b"} Jul 13 22:14:33.803446 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:14:33.803036 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-26w5h" event={"ID":"85265522-e114-4c04-b595-a26fcae86884","Type":"ContainerStarted","Data":"b14886d12f2eca32ccb5029b1e307d542ab13d4c31ce73d288dc213c753832ef"} Jul 13 22:14:33.819714 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:14:33.819650 2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-26w5h" podStartSLOduration=67.103568547 podStartE2EDuration="1m8.819634448s" totalImagesPullingTime="1.716065901s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-13 22:13:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 22:14:33.819090678 +0000 UTC m=+69.067006029" watchObservedRunningTime="2026-07-13 22:14:33.819634448 +0000 UTC m=+69.067549799" Jul 13 22:14:35.421294 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:14:35.421248 2583 patch_prober.go:28] interesting pod/image-registry-74f88c6495-7kk5p container/registry namespace/openshift-image-registry: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body={"errors":[{"code":"UNAVAILABLE","message":"service unavailable","detail":"health check failed: please see /debug/health"}]} Jul 13 22:14:35.421825 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:14:35.421319 2583 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-image-registry/image-registry-74f88c6495-7kk5p" podUID="2690217c-291d-4cbb-8ede-0ebe7802731c" containerName="registry" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 13 22:14:36.815794 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:14:36.815755 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-9hl5q" event={"ID":"d38c850d-d65b-4f99-ad24-5c1d443bff4b","Type":"ContainerStarted","Data":"cbdd5baf6162f620e77680cfa626fe38ea5bf09cc83057d135b93ac2e674569c"} Jul 13 22:14:36.817486 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:14:36.817454 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-v8mkx" event={"ID":"c8bbc3a9-6b5c-485d-b7ef-ca4cdca462ce","Type":"ContainerStarted","Data":"ddf0ae0c7029210b06d2037a764f2937ace6ae5f55f3805d90a3dcbe65ebc7fd"} Jul 13 22:14:36.817638 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:14:36.817514 2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-v8mkx" Jul 13 22:14:36.830156 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:14:36.830102 2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-9hl5q" podStartSLOduration=66.893238342 podStartE2EDuration="1m11.830081732s" totalImagesPullingTime="4.93684339s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-13 22:13:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 22:14:36.828978922 +0000 UTC m=+72.076894271" watchObservedRunningTime="2026-07-13 22:14:36.830081732 +0000 UTC m=+72.077997082" Jul 13 22:14:36.844695 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:14:36.844645 2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-v8mkx" podStartSLOduration=66.938056018 podStartE2EDuration="1m11.844628884s" totalImagesPullingTime="4.906572866s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-13 22:13:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 22:14:36.843265453 +0000 UTC m=+72.091180804" watchObservedRunningTime="2026-07-13 22:14:36.844628884 +0000 UTC m=+72.092544233" Jul 13 22:14:44.517264 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:14:44.517218 2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-9fx68"] Jul 13 22:14:44.520476 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:14:44.520455 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-9fx68" Jul 13 22:14:44.522776 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:14:44.522754 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\"" Jul 13 22:14:44.522917 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:14:44.522898 2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\"" Jul 13 22:14:44.523259 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:14:44.523241 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-q65jt\"" Jul 13 22:14:44.523800 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:14:44.523784 2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\"" Jul 13 22:14:44.523908 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:14:44.523892 2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\"" Jul 13 22:14:44.524052 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:14:44.523994 2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\"" Jul 13 22:14:44.524124 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:14:44.524091 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\"" Jul 13 22:14:44.621032 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:14:44.620994 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/d060f80c-835e-49e4-a012-e32fdbdb3890-node-exporter-tls\") pod \"node-exporter-9fx68\" (UID: \"d060f80c-835e-49e4-a012-e32fdbdb3890\") " pod="openshift-monitoring/node-exporter-9fx68" Jul 13 22:14:44.621223 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:14:44.621042 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/d060f80c-835e-49e4-a012-e32fdbdb3890-metrics-client-ca\") pod \"node-exporter-9fx68\" (UID: \"d060f80c-835e-49e4-a012-e32fdbdb3890\") " pod="openshift-monitoring/node-exporter-9fx68" Jul 13 22:14:44.621223 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:14:44.621125 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/d060f80c-835e-49e4-a012-e32fdbdb3890-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-9fx68\" (UID: \"d060f80c-835e-49e4-a012-e32fdbdb3890\") " pod="openshift-monitoring/node-exporter-9fx68" Jul 13 22:14:44.621223 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:14:44.621169 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/d060f80c-835e-49e4-a012-e32fdbdb3890-node-exporter-wtmp\") pod \"node-exporter-9fx68\" (UID: \"d060f80c-835e-49e4-a012-e32fdbdb3890\") " pod="openshift-monitoring/node-exporter-9fx68" Jul 13 22:14:44.621223 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:14:44.621190 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/d060f80c-835e-49e4-a012-e32fdbdb3890-node-exporter-textfile\") pod \"node-exporter-9fx68\" (UID: \"d060f80c-835e-49e4-a012-e32fdbdb3890\") " pod="openshift-monitoring/node-exporter-9fx68" Jul 13 22:14:44.621223 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:14:44.621214 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/d060f80c-835e-49e4-a012-e32fdbdb3890-node-exporter-accelerators-collector-config\") pod \"node-exporter-9fx68\" (UID: \"d060f80c-835e-49e4-a012-e32fdbdb3890\") " pod="openshift-monitoring/node-exporter-9fx68" Jul 13 22:14:44.621491 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:14:44.621232 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5xsds\" (UniqueName: \"kubernetes.io/projected/d060f80c-835e-49e4-a012-e32fdbdb3890-kube-api-access-5xsds\") pod \"node-exporter-9fx68\" (UID: \"d060f80c-835e-49e4-a012-e32fdbdb3890\") " pod="openshift-monitoring/node-exporter-9fx68" Jul 13 22:14:44.621491 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:14:44.621290 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/d060f80c-835e-49e4-a012-e32fdbdb3890-root\") pod \"node-exporter-9fx68\" (UID: \"d060f80c-835e-49e4-a012-e32fdbdb3890\") " pod="openshift-monitoring/node-exporter-9fx68" Jul 13 22:14:44.621491 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:14:44.621380 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/d060f80c-835e-49e4-a012-e32fdbdb3890-sys\") pod \"node-exporter-9fx68\" (UID: \"d060f80c-835e-49e4-a012-e32fdbdb3890\") " pod="openshift-monitoring/node-exporter-9fx68" Jul 13 22:14:44.722069 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:14:44.722009 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/d060f80c-835e-49e4-a012-e32fdbdb3890-node-exporter-wtmp\") pod \"node-exporter-9fx68\" (UID: \"d060f80c-835e-49e4-a012-e32fdbdb3890\") " pod="openshift-monitoring/node-exporter-9fx68" Jul 13 22:14:44.722069 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:14:44.722065 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/d060f80c-835e-49e4-a012-e32fdbdb3890-node-exporter-textfile\") pod \"node-exporter-9fx68\" (UID: \"d060f80c-835e-49e4-a012-e32fdbdb3890\") " pod="openshift-monitoring/node-exporter-9fx68" Jul 13 22:14:44.722325 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:14:44.722097 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/d060f80c-835e-49e4-a012-e32fdbdb3890-node-exporter-accelerators-collector-config\") pod \"node-exporter-9fx68\" (UID: \"d060f80c-835e-49e4-a012-e32fdbdb3890\") " pod="openshift-monitoring/node-exporter-9fx68" Jul 13 22:14:44.722325 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:14:44.722123 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5xsds\" (UniqueName: \"kubernetes.io/projected/d060f80c-835e-49e4-a012-e32fdbdb3890-kube-api-access-5xsds\") pod \"node-exporter-9fx68\" (UID: \"d060f80c-835e-49e4-a012-e32fdbdb3890\") " pod="openshift-monitoring/node-exporter-9fx68" Jul 13 22:14:44.722325 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:14:44.722148 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/d060f80c-835e-49e4-a012-e32fdbdb3890-root\") pod \"node-exporter-9fx68\" (UID: \"d060f80c-835e-49e4-a012-e32fdbdb3890\") " pod="openshift-monitoring/node-exporter-9fx68" Jul 13 22:14:44.722325 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:14:44.722182 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/d060f80c-835e-49e4-a012-e32fdbdb3890-sys\") pod \"node-exporter-9fx68\" (UID: \"d060f80c-835e-49e4-a012-e32fdbdb3890\") " pod="openshift-monitoring/node-exporter-9fx68" Jul 13 22:14:44.722325 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:14:44.722220 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/d060f80c-835e-49e4-a012-e32fdbdb3890-node-exporter-tls\") pod \"node-exporter-9fx68\" (UID: \"d060f80c-835e-49e4-a012-e32fdbdb3890\") " pod="openshift-monitoring/node-exporter-9fx68" Jul 13 22:14:44.722325 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:14:44.722249 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/d060f80c-835e-49e4-a012-e32fdbdb3890-metrics-client-ca\") pod \"node-exporter-9fx68\" (UID: \"d060f80c-835e-49e4-a012-e32fdbdb3890\") " pod="openshift-monitoring/node-exporter-9fx68" Jul 13 22:14:44.722325 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:14:44.722302 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/d060f80c-835e-49e4-a012-e32fdbdb3890-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-9fx68\" (UID: \"d060f80c-835e-49e4-a012-e32fdbdb3890\") " pod="openshift-monitoring/node-exporter-9fx68" Jul 13 22:14:44.722678 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:14:44.722430 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/d060f80c-835e-49e4-a012-e32fdbdb3890-node-exporter-textfile\") pod \"node-exporter-9fx68\" (UID: \"d060f80c-835e-49e4-a012-e32fdbdb3890\") " pod="openshift-monitoring/node-exporter-9fx68" Jul 13 22:14:44.722678 ip-10-0-133-179 kubenswrapper[2583]: E0713 22:14:44.722617 2583 secret.go:189] Couldn't get secret openshift-monitoring/node-exporter-tls: secret "node-exporter-tls" not found Jul 13 22:14:44.722678 ip-10-0-133-179 kubenswrapper[2583]: E0713 22:14:44.722678 2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d060f80c-835e-49e4-a012-e32fdbdb3890-node-exporter-tls podName:d060f80c-835e-49e4-a012-e32fdbdb3890 nodeName:}" failed. No retries permitted until 2026-07-13 22:14:45.222656603 +0000 UTC m=+80.470571935 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "node-exporter-tls" (UniqueName: "kubernetes.io/secret/d060f80c-835e-49e4-a012-e32fdbdb3890-node-exporter-tls") pod "node-exporter-9fx68" (UID: "d060f80c-835e-49e4-a012-e32fdbdb3890") : secret "node-exporter-tls" not found Jul 13 22:14:44.722830 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:14:44.722673 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/d060f80c-835e-49e4-a012-e32fdbdb3890-root\") pod \"node-exporter-9fx68\" (UID: \"d060f80c-835e-49e4-a012-e32fdbdb3890\") " pod="openshift-monitoring/node-exporter-9fx68" Jul 13 22:14:44.722830 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:14:44.722220 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/d060f80c-835e-49e4-a012-e32fdbdb3890-node-exporter-wtmp\") pod \"node-exporter-9fx68\" (UID: \"d060f80c-835e-49e4-a012-e32fdbdb3890\") " pod="openshift-monitoring/node-exporter-9fx68" Jul 13 22:14:44.722925 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:14:44.722878 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/d060f80c-835e-49e4-a012-e32fdbdb3890-sys\") pod \"node-exporter-9fx68\" (UID: \"d060f80c-835e-49e4-a012-e32fdbdb3890\") " pod="openshift-monitoring/node-exporter-9fx68" Jul 13 22:14:44.723024 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:14:44.722981 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/d060f80c-835e-49e4-a012-e32fdbdb3890-node-exporter-accelerators-collector-config\") pod \"node-exporter-9fx68\" (UID: \"d060f80c-835e-49e4-a012-e32fdbdb3890\") " pod="openshift-monitoring/node-exporter-9fx68" Jul 13 22:14:44.723202 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:14:44.723181 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/d060f80c-835e-49e4-a012-e32fdbdb3890-metrics-client-ca\") pod \"node-exporter-9fx68\" (UID: \"d060f80c-835e-49e4-a012-e32fdbdb3890\") " pod="openshift-monitoring/node-exporter-9fx68" Jul 13 22:14:44.724601 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:14:44.724564 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/d060f80c-835e-49e4-a012-e32fdbdb3890-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-9fx68\" (UID: \"d060f80c-835e-49e4-a012-e32fdbdb3890\") " pod="openshift-monitoring/node-exporter-9fx68" Jul 13 22:14:44.730132 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:14:44.730108 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5xsds\" (UniqueName: \"kubernetes.io/projected/d060f80c-835e-49e4-a012-e32fdbdb3890-kube-api-access-5xsds\") pod \"node-exporter-9fx68\" (UID: \"d060f80c-835e-49e4-a012-e32fdbdb3890\") " pod="openshift-monitoring/node-exporter-9fx68" Jul 13 22:14:45.226154 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:14:45.226109 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/d060f80c-835e-49e4-a012-e32fdbdb3890-node-exporter-tls\") pod \"node-exporter-9fx68\" (UID: \"d060f80c-835e-49e4-a012-e32fdbdb3890\") " pod="openshift-monitoring/node-exporter-9fx68" Jul 13 22:14:45.228435 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:14:45.228411 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/d060f80c-835e-49e4-a012-e32fdbdb3890-node-exporter-tls\") pod \"node-exporter-9fx68\" (UID: \"d060f80c-835e-49e4-a012-e32fdbdb3890\") " pod="openshift-monitoring/node-exporter-9fx68" Jul 13 22:14:45.419320 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:14:45.419288 2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-74f88c6495-7kk5p" Jul 13 22:14:45.430771 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:14:45.430738 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-9fx68" Jul 13 22:14:45.442421 ip-10-0-133-179 kubenswrapper[2583]: W0713 22:14:45.442391 2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd060f80c_835e_49e4_a012_e32fdbdb3890.slice/crio-efd4b59f910b49037cab973e29fb7aa3549a4b88dc5bb003b4334c8d26676674 WatchSource:0}: Error finding container efd4b59f910b49037cab973e29fb7aa3549a4b88dc5bb003b4334c8d26676674: Status 404 returned error can't find the container with id efd4b59f910b49037cab973e29fb7aa3549a4b88dc5bb003b4334c8d26676674 Jul 13 22:14:45.848415 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:14:45.848379 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-9fx68" event={"ID":"d060f80c-835e-49e4-a012-e32fdbdb3890","Type":"ContainerStarted","Data":"efd4b59f910b49037cab973e29fb7aa3549a4b88dc5bb003b4334c8d26676674"} Jul 13 22:14:46.853276 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:14:46.853239 2583 generic.go:356] "Generic (PLEG): container finished" podID="d060f80c-835e-49e4-a012-e32fdbdb3890" containerID="98a89db44988e404d0d2d947b63a2c883c3df8ed7450e57f6f7aadb14dd89bea" exitCode=0 Jul 13 22:14:46.853276 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:14:46.853280 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-9fx68" event={"ID":"d060f80c-835e-49e4-a012-e32fdbdb3890","Type":"ContainerDied","Data":"98a89db44988e404d0d2d947b63a2c883c3df8ed7450e57f6f7aadb14dd89bea"} Jul 13 22:14:47.765669 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:14:47.765641 2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-f7c956786-wvmw5" Jul 13 22:14:47.859958 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:14:47.859925 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-9fx68" event={"ID":"d060f80c-835e-49e4-a012-e32fdbdb3890","Type":"ContainerStarted","Data":"212c9487371d36d55048af91d51e9137bde36f5bd249575fc80737c88e7bb8ef"} Jul 13 22:14:47.860360 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:14:47.859972 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-9fx68" event={"ID":"d060f80c-835e-49e4-a012-e32fdbdb3890","Type":"ContainerStarted","Data":"e368bbdb081fa5b04354b7bc669efc6a745c0b62ce4b668ce74826f2f78b5816"} Jul 13 22:14:47.879719 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:14:47.879661 2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-9fx68" podStartSLOduration=2.754563551 podStartE2EDuration="3.879645118s" totalImagesPullingTime="1.059784366s" totalInitContainerRuntime="65.297201ms" isStatefulPod=true podCreationTimestamp="2026-07-13 22:14:44 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 22:14:47.878256016 +0000 UTC m=+83.126171365" watchObservedRunningTime="2026-07-13 22:14:47.879645118 +0000 UTC m=+83.127560518" Jul 13 22:14:50.433725 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:14:50.433681 2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-image-registry/image-registry-74f88c6495-7kk5p" podUID="2690217c-291d-4cbb-8ede-0ebe7802731c" containerName="registry" containerID="cri-o://406329a37adb98218fcda890e8ea3fd13d421dcce93ca7eb3db39be5fc1c58bd" gracePeriod=30 Jul 13 22:14:50.674410 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:14:50.674387 2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-74f88c6495-7kk5p" Jul 13 22:14:50.767456 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:14:50.767375 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/2690217c-291d-4cbb-8ede-0ebe7802731c-registry-certificates\") pod \"2690217c-291d-4cbb-8ede-0ebe7802731c\" (UID: \"2690217c-291d-4cbb-8ede-0ebe7802731c\") " Jul 13 22:14:50.767456 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:14:50.767429 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6lwmw\" (UniqueName: \"kubernetes.io/projected/2690217c-291d-4cbb-8ede-0ebe7802731c-kube-api-access-6lwmw\") pod \"2690217c-291d-4cbb-8ede-0ebe7802731c\" (UID: \"2690217c-291d-4cbb-8ede-0ebe7802731c\") " Jul 13 22:14:50.767684 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:14:50.767466 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/2690217c-291d-4cbb-8ede-0ebe7802731c-trusted-ca\") pod \"2690217c-291d-4cbb-8ede-0ebe7802731c\" (UID: \"2690217c-291d-4cbb-8ede-0ebe7802731c\") " Jul 13 22:14:50.767684 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:14:50.767497 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/2690217c-291d-4cbb-8ede-0ebe7802731c-ca-trust-extracted\") pod \"2690217c-291d-4cbb-8ede-0ebe7802731c\" (UID: \"2690217c-291d-4cbb-8ede-0ebe7802731c\") " Jul 13 22:14:50.767684 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:14:50.767531 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/2690217c-291d-4cbb-8ede-0ebe7802731c-installation-pull-secrets\") pod \"2690217c-291d-4cbb-8ede-0ebe7802731c\" (UID: \"2690217c-291d-4cbb-8ede-0ebe7802731c\") " Jul 13 22:14:50.767684 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:14:50.767563 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/2690217c-291d-4cbb-8ede-0ebe7802731c-registry-tls\") pod \"2690217c-291d-4cbb-8ede-0ebe7802731c\" (UID: \"2690217c-291d-4cbb-8ede-0ebe7802731c\") " Jul 13 22:14:50.767684 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:14:50.767617 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/2690217c-291d-4cbb-8ede-0ebe7802731c-image-registry-private-configuration\") pod \"2690217c-291d-4cbb-8ede-0ebe7802731c\" (UID: \"2690217c-291d-4cbb-8ede-0ebe7802731c\") " Jul 13 22:14:50.767684 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:14:50.767649 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/2690217c-291d-4cbb-8ede-0ebe7802731c-bound-sa-token\") pod \"2690217c-291d-4cbb-8ede-0ebe7802731c\" (UID: \"2690217c-291d-4cbb-8ede-0ebe7802731c\") " Jul 13 22:14:50.768024 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:14:50.767992 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2690217c-291d-4cbb-8ede-0ebe7802731c-registry-certificates" (OuterVolumeSpecName: "registry-certificates") pod "2690217c-291d-4cbb-8ede-0ebe7802731c" (UID: "2690217c-291d-4cbb-8ede-0ebe7802731c"). InnerVolumeSpecName "registry-certificates". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 13 22:14:50.768101 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:14:50.768016 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2690217c-291d-4cbb-8ede-0ebe7802731c-trusted-ca" (OuterVolumeSpecName: "trusted-ca") pod "2690217c-291d-4cbb-8ede-0ebe7802731c" (UID: "2690217c-291d-4cbb-8ede-0ebe7802731c"). InnerVolumeSpecName "trusted-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 13 22:14:50.770476 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:14:50.770445 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2690217c-291d-4cbb-8ede-0ebe7802731c-kube-api-access-6lwmw" (OuterVolumeSpecName: "kube-api-access-6lwmw") pod "2690217c-291d-4cbb-8ede-0ebe7802731c" (UID: "2690217c-291d-4cbb-8ede-0ebe7802731c"). InnerVolumeSpecName "kube-api-access-6lwmw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 13 22:14:50.770606 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:14:50.770471 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2690217c-291d-4cbb-8ede-0ebe7802731c-installation-pull-secrets" (OuterVolumeSpecName: "installation-pull-secrets") pod "2690217c-291d-4cbb-8ede-0ebe7802731c" (UID: "2690217c-291d-4cbb-8ede-0ebe7802731c"). InnerVolumeSpecName "installation-pull-secrets". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 13 22:14:50.770675 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:14:50.770608 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2690217c-291d-4cbb-8ede-0ebe7802731c-registry-tls" (OuterVolumeSpecName: "registry-tls") pod "2690217c-291d-4cbb-8ede-0ebe7802731c" (UID: "2690217c-291d-4cbb-8ede-0ebe7802731c"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 13 22:14:50.770835 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:14:50.770808 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2690217c-291d-4cbb-8ede-0ebe7802731c-image-registry-private-configuration" (OuterVolumeSpecName: "image-registry-private-configuration") pod "2690217c-291d-4cbb-8ede-0ebe7802731c" (UID: "2690217c-291d-4cbb-8ede-0ebe7802731c"). InnerVolumeSpecName "image-registry-private-configuration". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 13 22:14:50.770954 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:14:50.770839 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2690217c-291d-4cbb-8ede-0ebe7802731c-bound-sa-token" (OuterVolumeSpecName: "bound-sa-token") pod "2690217c-291d-4cbb-8ede-0ebe7802731c" (UID: "2690217c-291d-4cbb-8ede-0ebe7802731c"). InnerVolumeSpecName "bound-sa-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 13 22:14:50.777597 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:14:50.777553 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2690217c-291d-4cbb-8ede-0ebe7802731c-ca-trust-extracted" (OuterVolumeSpecName: "ca-trust-extracted") pod "2690217c-291d-4cbb-8ede-0ebe7802731c" (UID: "2690217c-291d-4cbb-8ede-0ebe7802731c"). InnerVolumeSpecName "ca-trust-extracted". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 13 22:14:50.868285 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:14:50.868251 2583 reconciler_common.go:299] "Volume detached for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/2690217c-291d-4cbb-8ede-0ebe7802731c-registry-certificates\") on node \"ip-10-0-133-179.ec2.internal\" DevicePath \"\"" Jul 13 22:14:50.868285 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:14:50.868275 2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6lwmw\" (UniqueName: \"kubernetes.io/projected/2690217c-291d-4cbb-8ede-0ebe7802731c-kube-api-access-6lwmw\") on node \"ip-10-0-133-179.ec2.internal\" DevicePath \"\"" Jul 13 22:14:50.868285 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:14:50.868285 2583 reconciler_common.go:299] "Volume detached for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/2690217c-291d-4cbb-8ede-0ebe7802731c-trusted-ca\") on node \"ip-10-0-133-179.ec2.internal\" DevicePath \"\"" Jul 13 22:14:50.868511 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:14:50.868295 2583 reconciler_common.go:299] "Volume detached for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/2690217c-291d-4cbb-8ede-0ebe7802731c-ca-trust-extracted\") on node \"ip-10-0-133-179.ec2.internal\" DevicePath \"\"" Jul 13 22:14:50.868511 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:14:50.868305 2583 reconciler_common.go:299] "Volume detached for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/2690217c-291d-4cbb-8ede-0ebe7802731c-installation-pull-secrets\") on node \"ip-10-0-133-179.ec2.internal\" DevicePath \"\"" Jul 13 22:14:50.868511 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:14:50.868313 2583 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/2690217c-291d-4cbb-8ede-0ebe7802731c-registry-tls\") on node \"ip-10-0-133-179.ec2.internal\" DevicePath \"\"" Jul 13 22:14:50.868511 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:14:50.868322 2583 reconciler_common.go:299] "Volume detached for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/2690217c-291d-4cbb-8ede-0ebe7802731c-image-registry-private-configuration\") on node \"ip-10-0-133-179.ec2.internal\" DevicePath \"\"" Jul 13 22:14:50.868511 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:14:50.868331 2583 reconciler_common.go:299] "Volume detached for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/2690217c-291d-4cbb-8ede-0ebe7802731c-bound-sa-token\") on node \"ip-10-0-133-179.ec2.internal\" DevicePath \"\"" Jul 13 22:14:50.871024 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:14:50.870993 2583 generic.go:356] "Generic (PLEG): container finished" podID="2690217c-291d-4cbb-8ede-0ebe7802731c" containerID="406329a37adb98218fcda890e8ea3fd13d421dcce93ca7eb3db39be5fc1c58bd" exitCode=0 Jul 13 22:14:50.871143 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:14:50.871037 2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-74f88c6495-7kk5p" Jul 13 22:14:50.871143 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:14:50.871080 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-74f88c6495-7kk5p" event={"ID":"2690217c-291d-4cbb-8ede-0ebe7802731c","Type":"ContainerDied","Data":"406329a37adb98218fcda890e8ea3fd13d421dcce93ca7eb3db39be5fc1c58bd"} Jul 13 22:14:50.871143 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:14:50.871139 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-74f88c6495-7kk5p" event={"ID":"2690217c-291d-4cbb-8ede-0ebe7802731c","Type":"ContainerDied","Data":"9fd137bfdb99741f9a2b4744b86a2e7fa43f9e5689dc30ee1bad10f9a55910c9"} Jul 13 22:14:50.871310 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:14:50.871167 2583 scope.go:117] "RemoveContainer" containerID="406329a37adb98218fcda890e8ea3fd13d421dcce93ca7eb3db39be5fc1c58bd" Jul 13 22:14:50.880030 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:14:50.880009 2583 scope.go:117] "RemoveContainer" containerID="406329a37adb98218fcda890e8ea3fd13d421dcce93ca7eb3db39be5fc1c58bd" Jul 13 22:14:50.880292 ip-10-0-133-179 kubenswrapper[2583]: E0713 22:14:50.880271 2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"406329a37adb98218fcda890e8ea3fd13d421dcce93ca7eb3db39be5fc1c58bd\": container with ID starting with 406329a37adb98218fcda890e8ea3fd13d421dcce93ca7eb3db39be5fc1c58bd not found: ID does not exist" containerID="406329a37adb98218fcda890e8ea3fd13d421dcce93ca7eb3db39be5fc1c58bd" Jul 13 22:14:50.880347 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:14:50.880303 2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"406329a37adb98218fcda890e8ea3fd13d421dcce93ca7eb3db39be5fc1c58bd"} err="failed to get container status \"406329a37adb98218fcda890e8ea3fd13d421dcce93ca7eb3db39be5fc1c58bd\": rpc error: code = NotFound desc = could not find container \"406329a37adb98218fcda890e8ea3fd13d421dcce93ca7eb3db39be5fc1c58bd\": container with ID starting with 406329a37adb98218fcda890e8ea3fd13d421dcce93ca7eb3db39be5fc1c58bd not found: ID does not exist" Jul 13 22:14:50.895339 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:14:50.895315 2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-74f88c6495-7kk5p"] Jul 13 22:14:50.903412 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:14:50.903393 2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-image-registry/image-registry-74f88c6495-7kk5p"] Jul 13 22:14:51.388678 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:14:51.388648 2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2690217c-291d-4cbb-8ede-0ebe7802731c" path="/var/lib/kubelet/pods/2690217c-291d-4cbb-8ede-0ebe7802731c/volumes" Jul 13 22:15:07.824286 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:15:07.824251 2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-v8mkx" Jul 13 22:18:24.415539 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:18:24.415502 2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-lws-operator/lws-controller-manager-84865b6cc-mqgj2"] Jul 13 22:18:24.415971 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:18:24.415770 2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2690217c-291d-4cbb-8ede-0ebe7802731c" containerName="registry" Jul 13 22:18:24.415971 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:18:24.415781 2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="2690217c-291d-4cbb-8ede-0ebe7802731c" containerName="registry" Jul 13 22:18:24.415971 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:18:24.415832 2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="2690217c-291d-4cbb-8ede-0ebe7802731c" containerName="registry" Jul 13 22:18:24.418609 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:18:24.418591 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-84865b6cc-mqgj2" Jul 13 22:18:24.420982 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:18:24.420959 2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"openshift-service-ca.crt\"" Jul 13 22:18:24.421096 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:18:24.421031 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"webhook-server-cert\"" Jul 13 22:18:24.421702 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:18:24.421688 2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"kube-root-ca.crt\"" Jul 13 22:18:24.421796 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:18:24.421763 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"lws-controller-manager-dockercfg-6fbvd\"" Jul 13 22:18:24.422629 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:18:24.422615 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"metrics-server-cert\"" Jul 13 22:18:24.424738 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:18:24.424722 2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"lws-manager-config\"" Jul 13 22:18:24.432739 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:18:24.432717 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-84865b6cc-mqgj2"] Jul 13 22:18:24.482440 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:18:24.482412 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/b42903c8-449f-4b60-8749-79e3ffeb03b7-metrics-cert\") pod \"lws-controller-manager-84865b6cc-mqgj2\" (UID: \"b42903c8-449f-4b60-8749-79e3ffeb03b7\") " pod="openshift-lws-operator/lws-controller-manager-84865b6cc-mqgj2" Jul 13 22:18:24.482633 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:18:24.482460 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/b42903c8-449f-4b60-8749-79e3ffeb03b7-manager-config\") pod \"lws-controller-manager-84865b6cc-mqgj2\" (UID: \"b42903c8-449f-4b60-8749-79e3ffeb03b7\") " pod="openshift-lws-operator/lws-controller-manager-84865b6cc-mqgj2" Jul 13 22:18:24.482633 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:18:24.482482 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/b42903c8-449f-4b60-8749-79e3ffeb03b7-cert\") pod \"lws-controller-manager-84865b6cc-mqgj2\" (UID: \"b42903c8-449f-4b60-8749-79e3ffeb03b7\") " pod="openshift-lws-operator/lws-controller-manager-84865b6cc-mqgj2" Jul 13 22:18:24.482633 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:18:24.482604 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rqrc4\" (UniqueName: \"kubernetes.io/projected/b42903c8-449f-4b60-8749-79e3ffeb03b7-kube-api-access-rqrc4\") pod \"lws-controller-manager-84865b6cc-mqgj2\" (UID: \"b42903c8-449f-4b60-8749-79e3ffeb03b7\") " pod="openshift-lws-operator/lws-controller-manager-84865b6cc-mqgj2" Jul 13 22:18:24.583911 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:18:24.583877 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/b42903c8-449f-4b60-8749-79e3ffeb03b7-manager-config\") pod \"lws-controller-manager-84865b6cc-mqgj2\" (UID: \"b42903c8-449f-4b60-8749-79e3ffeb03b7\") " pod="openshift-lws-operator/lws-controller-manager-84865b6cc-mqgj2" Jul 13 22:18:24.584110 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:18:24.583916 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/b42903c8-449f-4b60-8749-79e3ffeb03b7-cert\") pod \"lws-controller-manager-84865b6cc-mqgj2\" (UID: \"b42903c8-449f-4b60-8749-79e3ffeb03b7\") " pod="openshift-lws-operator/lws-controller-manager-84865b6cc-mqgj2" Jul 13 22:18:24.584110 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:18:24.583989 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rqrc4\" (UniqueName: \"kubernetes.io/projected/b42903c8-449f-4b60-8749-79e3ffeb03b7-kube-api-access-rqrc4\") pod \"lws-controller-manager-84865b6cc-mqgj2\" (UID: \"b42903c8-449f-4b60-8749-79e3ffeb03b7\") " pod="openshift-lws-operator/lws-controller-manager-84865b6cc-mqgj2" Jul 13 22:18:24.584110 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:18:24.584026 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/b42903c8-449f-4b60-8749-79e3ffeb03b7-metrics-cert\") pod \"lws-controller-manager-84865b6cc-mqgj2\" (UID: \"b42903c8-449f-4b60-8749-79e3ffeb03b7\") " pod="openshift-lws-operator/lws-controller-manager-84865b6cc-mqgj2" Jul 13 22:18:24.584647 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:18:24.584620 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/b42903c8-449f-4b60-8749-79e3ffeb03b7-manager-config\") pod \"lws-controller-manager-84865b6cc-mqgj2\" (UID: \"b42903c8-449f-4b60-8749-79e3ffeb03b7\") " pod="openshift-lws-operator/lws-controller-manager-84865b6cc-mqgj2" Jul 13 22:18:24.586403 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:18:24.586371 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/b42903c8-449f-4b60-8749-79e3ffeb03b7-metrics-cert\") pod \"lws-controller-manager-84865b6cc-mqgj2\" (UID: \"b42903c8-449f-4b60-8749-79e3ffeb03b7\") " pod="openshift-lws-operator/lws-controller-manager-84865b6cc-mqgj2" Jul 13 22:18:24.586502 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:18:24.586417 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/b42903c8-449f-4b60-8749-79e3ffeb03b7-cert\") pod \"lws-controller-manager-84865b6cc-mqgj2\" (UID: \"b42903c8-449f-4b60-8749-79e3ffeb03b7\") " pod="openshift-lws-operator/lws-controller-manager-84865b6cc-mqgj2" Jul 13 22:18:24.604340 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:18:24.604312 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rqrc4\" (UniqueName: \"kubernetes.io/projected/b42903c8-449f-4b60-8749-79e3ffeb03b7-kube-api-access-rqrc4\") pod \"lws-controller-manager-84865b6cc-mqgj2\" (UID: \"b42903c8-449f-4b60-8749-79e3ffeb03b7\") " pod="openshift-lws-operator/lws-controller-manager-84865b6cc-mqgj2" Jul 13 22:18:24.728082 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:18:24.727995 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-84865b6cc-mqgj2" Jul 13 22:18:24.856664 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:18:24.856487 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-84865b6cc-mqgj2"] Jul 13 22:18:24.859141 ip-10-0-133-179 kubenswrapper[2583]: W0713 22:18:24.859109 2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb42903c8_449f_4b60_8749_79e3ffeb03b7.slice/crio-fbb16f85cefb3f258d4fbd71eec61c42638ca56db8aa8ecb891b211ddbd9f910 WatchSource:0}: Error finding container fbb16f85cefb3f258d4fbd71eec61c42638ca56db8aa8ecb891b211ddbd9f910: Status 404 returned error can't find the container with id fbb16f85cefb3f258d4fbd71eec61c42638ca56db8aa8ecb891b211ddbd9f910 Jul 13 22:18:25.202063 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:18:25.202036 2583 kubelet.go:1648] "Image garbage collection succeeded" Jul 13 22:18:25.570138 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:18:25.570097 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-84865b6cc-mqgj2" event={"ID":"b42903c8-449f-4b60-8749-79e3ffeb03b7","Type":"ContainerStarted","Data":"fbb16f85cefb3f258d4fbd71eec61c42638ca56db8aa8ecb891b211ddbd9f910"} Jul 13 22:18:27.578625 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:18:27.578555 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-84865b6cc-mqgj2" event={"ID":"b42903c8-449f-4b60-8749-79e3ffeb03b7","Type":"ContainerStarted","Data":"69434a326f27f15092de3a2c73acf35a502313d429504afd695ceddee5649e58"} Jul 13 22:18:27.579027 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:18:27.578681 2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-lws-operator/lws-controller-manager-84865b6cc-mqgj2" Jul 13 22:18:27.600432 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:18:27.600383 2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-lws-operator/lws-controller-manager-84865b6cc-mqgj2" podStartSLOduration=1.208171151 podStartE2EDuration="3.600366838s" totalImagesPullingTime="2.392195687s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-13 22:18:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 22:18:27.599013762 +0000 UTC m=+302.846929113" watchObservedRunningTime="2026-07-13 22:18:27.600366838 +0000 UTC m=+302.848282188" Jul 13 22:18:27.652234 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:18:27.652193 2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-d5bf5468b-bmrcj"] Jul 13 22:18:27.655404 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:18:27.655376 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-d5bf5468b-bmrcj" Jul 13 22:18:27.658522 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:18:27.658493 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-webhook-cert\"" Jul 13 22:18:27.659198 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:18:27.658714 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-service-cert\"" Jul 13 22:18:27.659198 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:18:27.658999 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-dockercfg-2l2fh\"" Jul 13 22:18:27.659684 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:18:27.659663 2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\"" Jul 13 22:18:27.660280 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:18:27.660254 2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\"" Jul 13 22:18:27.673785 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:18:27.673763 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-d5bf5468b-bmrcj"] Jul 13 22:18:27.810091 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:18:27.810061 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/9189e9b0-0cba-43f0-84a7-4a1024b285b0-apiservice-cert\") pod \"opendatahub-operator-controller-manager-d5bf5468b-bmrcj\" (UID: \"9189e9b0-0cba-43f0-84a7-4a1024b285b0\") " pod="opendatahub/opendatahub-operator-controller-manager-d5bf5468b-bmrcj" Jul 13 22:18:27.810091 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:18:27.810091 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/9189e9b0-0cba-43f0-84a7-4a1024b285b0-webhook-cert\") pod \"opendatahub-operator-controller-manager-d5bf5468b-bmrcj\" (UID: \"9189e9b0-0cba-43f0-84a7-4a1024b285b0\") " pod="opendatahub/opendatahub-operator-controller-manager-d5bf5468b-bmrcj" Jul 13 22:18:27.810283 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:18:27.810117 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/9189e9b0-0cba-43f0-84a7-4a1024b285b0-manifests\") pod \"opendatahub-operator-controller-manager-d5bf5468b-bmrcj\" (UID: \"9189e9b0-0cba-43f0-84a7-4a1024b285b0\") " pod="opendatahub/opendatahub-operator-controller-manager-d5bf5468b-bmrcj" Jul 13 22:18:27.810283 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:18:27.810193 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jpmxw\" (UniqueName: \"kubernetes.io/projected/9189e9b0-0cba-43f0-84a7-4a1024b285b0-kube-api-access-jpmxw\") pod \"opendatahub-operator-controller-manager-d5bf5468b-bmrcj\" (UID: \"9189e9b0-0cba-43f0-84a7-4a1024b285b0\") " pod="opendatahub/opendatahub-operator-controller-manager-d5bf5468b-bmrcj" Jul 13 22:18:27.810283 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:18:27.810235 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/9189e9b0-0cba-43f0-84a7-4a1024b285b0-tmp\") pod \"opendatahub-operator-controller-manager-d5bf5468b-bmrcj\" (UID: \"9189e9b0-0cba-43f0-84a7-4a1024b285b0\") " pod="opendatahub/opendatahub-operator-controller-manager-d5bf5468b-bmrcj" Jul 13 22:18:27.911640 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:18:27.911535 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jpmxw\" (UniqueName: \"kubernetes.io/projected/9189e9b0-0cba-43f0-84a7-4a1024b285b0-kube-api-access-jpmxw\") pod \"opendatahub-operator-controller-manager-d5bf5468b-bmrcj\" (UID: \"9189e9b0-0cba-43f0-84a7-4a1024b285b0\") " pod="opendatahub/opendatahub-operator-controller-manager-d5bf5468b-bmrcj" Jul 13 22:18:27.911640 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:18:27.911603 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/9189e9b0-0cba-43f0-84a7-4a1024b285b0-tmp\") pod \"opendatahub-operator-controller-manager-d5bf5468b-bmrcj\" (UID: \"9189e9b0-0cba-43f0-84a7-4a1024b285b0\") " pod="opendatahub/opendatahub-operator-controller-manager-d5bf5468b-bmrcj" Jul 13 22:18:27.911817 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:18:27.911651 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/9189e9b0-0cba-43f0-84a7-4a1024b285b0-apiservice-cert\") pod \"opendatahub-operator-controller-manager-d5bf5468b-bmrcj\" (UID: \"9189e9b0-0cba-43f0-84a7-4a1024b285b0\") " pod="opendatahub/opendatahub-operator-controller-manager-d5bf5468b-bmrcj" Jul 13 22:18:27.911817 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:18:27.911668 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/9189e9b0-0cba-43f0-84a7-4a1024b285b0-webhook-cert\") pod \"opendatahub-operator-controller-manager-d5bf5468b-bmrcj\" (UID: \"9189e9b0-0cba-43f0-84a7-4a1024b285b0\") " pod="opendatahub/opendatahub-operator-controller-manager-d5bf5468b-bmrcj" Jul 13 22:18:27.911817 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:18:27.911690 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/9189e9b0-0cba-43f0-84a7-4a1024b285b0-manifests\") pod \"opendatahub-operator-controller-manager-d5bf5468b-bmrcj\" (UID: \"9189e9b0-0cba-43f0-84a7-4a1024b285b0\") " pod="opendatahub/opendatahub-operator-controller-manager-d5bf5468b-bmrcj" Jul 13 22:18:27.912125 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:18:27.912098 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/9189e9b0-0cba-43f0-84a7-4a1024b285b0-manifests\") pod \"opendatahub-operator-controller-manager-d5bf5468b-bmrcj\" (UID: \"9189e9b0-0cba-43f0-84a7-4a1024b285b0\") " pod="opendatahub/opendatahub-operator-controller-manager-d5bf5468b-bmrcj" Jul 13 22:18:27.913968 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:18:27.913947 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/9189e9b0-0cba-43f0-84a7-4a1024b285b0-tmp\") pod \"opendatahub-operator-controller-manager-d5bf5468b-bmrcj\" (UID: \"9189e9b0-0cba-43f0-84a7-4a1024b285b0\") " pod="opendatahub/opendatahub-operator-controller-manager-d5bf5468b-bmrcj" Jul 13 22:18:27.914093 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:18:27.914076 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/9189e9b0-0cba-43f0-84a7-4a1024b285b0-apiservice-cert\") pod \"opendatahub-operator-controller-manager-d5bf5468b-bmrcj\" (UID: \"9189e9b0-0cba-43f0-84a7-4a1024b285b0\") " pod="opendatahub/opendatahub-operator-controller-manager-d5bf5468b-bmrcj" Jul 13 22:18:27.914177 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:18:27.914156 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/9189e9b0-0cba-43f0-84a7-4a1024b285b0-webhook-cert\") pod \"opendatahub-operator-controller-manager-d5bf5468b-bmrcj\" (UID: \"9189e9b0-0cba-43f0-84a7-4a1024b285b0\") " pod="opendatahub/opendatahub-operator-controller-manager-d5bf5468b-bmrcj" Jul 13 22:18:27.920074 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:18:27.920050 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jpmxw\" (UniqueName: \"kubernetes.io/projected/9189e9b0-0cba-43f0-84a7-4a1024b285b0-kube-api-access-jpmxw\") pod \"opendatahub-operator-controller-manager-d5bf5468b-bmrcj\" (UID: \"9189e9b0-0cba-43f0-84a7-4a1024b285b0\") " pod="opendatahub/opendatahub-operator-controller-manager-d5bf5468b-bmrcj" Jul 13 22:18:27.965871 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:18:27.965839 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-d5bf5468b-bmrcj" Jul 13 22:18:28.102102 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:18:28.102065 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-d5bf5468b-bmrcj"] Jul 13 22:18:28.108333 ip-10-0-133-179 kubenswrapper[2583]: W0713 22:18:28.108304 2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9189e9b0_0cba_43f0_84a7_4a1024b285b0.slice/crio-04bd032b50968e68378598f53b010d8d3ed5392a3138a230243aef3eccb3fdab WatchSource:0}: Error finding container 04bd032b50968e68378598f53b010d8d3ed5392a3138a230243aef3eccb3fdab: Status 404 returned error can't find the container with id 04bd032b50968e68378598f53b010d8d3ed5392a3138a230243aef3eccb3fdab Jul 13 22:18:28.110060 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:18:28.110046 2583 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 13 22:18:28.583940 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:18:28.583907 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-d5bf5468b-bmrcj" event={"ID":"9189e9b0-0cba-43f0-84a7-4a1024b285b0","Type":"ContainerStarted","Data":"04bd032b50968e68378598f53b010d8d3ed5392a3138a230243aef3eccb3fdab"} Jul 13 22:18:31.596855 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:18:31.596819 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-d5bf5468b-bmrcj" event={"ID":"9189e9b0-0cba-43f0-84a7-4a1024b285b0","Type":"ContainerStarted","Data":"ec844afe06350337cd060ae15d17c610b5b64d3695d55e111cb6f89229ea2523"} Jul 13 22:18:32.601625 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:18:32.601564 2583 generic.go:356] "Generic (PLEG): container finished" podID="9189e9b0-0cba-43f0-84a7-4a1024b285b0" containerID="ec844afe06350337cd060ae15d17c610b5b64d3695d55e111cb6f89229ea2523" exitCode=0 Jul 13 22:18:32.601625 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:18:32.601608 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-d5bf5468b-bmrcj" event={"ID":"9189e9b0-0cba-43f0-84a7-4a1024b285b0","Type":"ContainerDied","Data":"ec844afe06350337cd060ae15d17c610b5b64d3695d55e111cb6f89229ea2523"} Jul 13 22:18:33.606635 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:18:33.606598 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-d5bf5468b-bmrcj" event={"ID":"9189e9b0-0cba-43f0-84a7-4a1024b285b0","Type":"ContainerStarted","Data":"180f39325792d862f2ea07adf0e79a82d6b42bdd7e2fb5c739c05463064660ca"} Jul 13 22:18:33.607059 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:18:33.606713 2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/opendatahub-operator-controller-manager-d5bf5468b-bmrcj" Jul 13 22:18:33.630102 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:18:33.630045 2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/opendatahub-operator-controller-manager-d5bf5468b-bmrcj" podStartSLOduration=2.336252026 podStartE2EDuration="6.630023537s" totalImagesPullingTime="3.222333785s" totalInitContainerRuntime="1.071437726s" isStatefulPod=false podCreationTimestamp="2026-07-13 22:18:27 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 22:18:33.628388763 +0000 UTC m=+308.876304125" watchObservedRunningTime="2026-07-13 22:18:33.630023537 +0000 UTC m=+308.877938887" Jul 13 22:18:38.586735 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:18:38.586698 2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-lws-operator/lws-controller-manager-84865b6cc-mqgj2" Jul 13 22:18:44.612686 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:18:44.612658 2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/opendatahub-operator-controller-manager-d5bf5468b-bmrcj" Jul 13 22:18:56.106769 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:18:56.106736 2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-7f964fb678lhs8d"] Jul 13 22:18:56.111381 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:18:56.111357 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7f964fb678lhs8d" Jul 13 22:18:56.113838 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:18:56.113813 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"data-science-gateway-data-science-gateway-class-dockercfg-6v4rp\"" Jul 13 22:18:56.114790 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:18:56.114766 2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\"" Jul 13 22:18:56.114884 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:18:56.114807 2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\"" Jul 13 22:18:56.115231 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:18:56.115212 2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\"" Jul 13 22:18:56.123271 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:18:56.122944 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/318be4cb-7ac5-4039-a1be-f26e6f9f4d18-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-7f964fb678lhs8d\" (UID: \"318be4cb-7ac5-4039-a1be-f26e6f9f4d18\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7f964fb678lhs8d" Jul 13 22:18:56.123391 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:18:56.123322 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/318be4cb-7ac5-4039-a1be-f26e6f9f4d18-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-7f964fb678lhs8d\" (UID: \"318be4cb-7ac5-4039-a1be-f26e6f9f4d18\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7f964fb678lhs8d" Jul 13 22:18:56.123391 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:18:56.123364 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/318be4cb-7ac5-4039-a1be-f26e6f9f4d18-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-7f964fb678lhs8d\" (UID: \"318be4cb-7ac5-4039-a1be-f26e6f9f4d18\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7f964fb678lhs8d" Jul 13 22:18:56.123504 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:18:56.123416 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gb5xh\" (UniqueName: \"kubernetes.io/projected/318be4cb-7ac5-4039-a1be-f26e6f9f4d18-kube-api-access-gb5xh\") pod \"data-science-gateway-data-science-gateway-class-7f964fb678lhs8d\" (UID: \"318be4cb-7ac5-4039-a1be-f26e6f9f4d18\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7f964fb678lhs8d" Jul 13 22:18:56.123604 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:18:56.123585 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/318be4cb-7ac5-4039-a1be-f26e6f9f4d18-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-7f964fb678lhs8d\" (UID: \"318be4cb-7ac5-4039-a1be-f26e6f9f4d18\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7f964fb678lhs8d" Jul 13 22:18:56.123664 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:18:56.123636 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/318be4cb-7ac5-4039-a1be-f26e6f9f4d18-istio-data\") pod \"data-science-gateway-data-science-gateway-class-7f964fb678lhs8d\" (UID: \"318be4cb-7ac5-4039-a1be-f26e6f9f4d18\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7f964fb678lhs8d" Jul 13 22:18:56.123717 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:18:56.123668 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/318be4cb-7ac5-4039-a1be-f26e6f9f4d18-istio-token\") pod \"data-science-gateway-data-science-gateway-class-7f964fb678lhs8d\" (UID: \"318be4cb-7ac5-4039-a1be-f26e6f9f4d18\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7f964fb678lhs8d" Jul 13 22:18:56.123770 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:18:56.123711 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/318be4cb-7ac5-4039-a1be-f26e6f9f4d18-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-7f964fb678lhs8d\" (UID: \"318be4cb-7ac5-4039-a1be-f26e6f9f4d18\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7f964fb678lhs8d" Jul 13 22:18:56.123823 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:18:56.123770 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/318be4cb-7ac5-4039-a1be-f26e6f9f4d18-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-7f964fb678lhs8d\" (UID: \"318be4cb-7ac5-4039-a1be-f26e6f9f4d18\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7f964fb678lhs8d" Jul 13 22:18:56.155161 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:18:56.155124 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-7f964fb678lhs8d"] Jul 13 22:18:56.224641 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:18:56.224601 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/318be4cb-7ac5-4039-a1be-f26e6f9f4d18-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-7f964fb678lhs8d\" (UID: \"318be4cb-7ac5-4039-a1be-f26e6f9f4d18\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7f964fb678lhs8d" Jul 13 22:18:56.224829 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:18:56.224649 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/318be4cb-7ac5-4039-a1be-f26e6f9f4d18-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-7f964fb678lhs8d\" (UID: \"318be4cb-7ac5-4039-a1be-f26e6f9f4d18\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7f964fb678lhs8d" Jul 13 22:18:56.224829 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:18:56.224674 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/318be4cb-7ac5-4039-a1be-f26e6f9f4d18-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-7f964fb678lhs8d\" (UID: \"318be4cb-7ac5-4039-a1be-f26e6f9f4d18\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7f964fb678lhs8d" Jul 13 22:18:56.224829 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:18:56.224692 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/318be4cb-7ac5-4039-a1be-f26e6f9f4d18-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-7f964fb678lhs8d\" (UID: \"318be4cb-7ac5-4039-a1be-f26e6f9f4d18\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7f964fb678lhs8d" Jul 13 22:18:56.224829 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:18:56.224730 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gb5xh\" (UniqueName: \"kubernetes.io/projected/318be4cb-7ac5-4039-a1be-f26e6f9f4d18-kube-api-access-gb5xh\") pod \"data-science-gateway-data-science-gateway-class-7f964fb678lhs8d\" (UID: \"318be4cb-7ac5-4039-a1be-f26e6f9f4d18\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7f964fb678lhs8d" Jul 13 22:18:56.224829 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:18:56.224755 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/318be4cb-7ac5-4039-a1be-f26e6f9f4d18-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-7f964fb678lhs8d\" (UID: \"318be4cb-7ac5-4039-a1be-f26e6f9f4d18\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7f964fb678lhs8d" Jul 13 22:18:56.224829 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:18:56.224787 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/318be4cb-7ac5-4039-a1be-f26e6f9f4d18-istio-data\") pod \"data-science-gateway-data-science-gateway-class-7f964fb678lhs8d\" (UID: \"318be4cb-7ac5-4039-a1be-f26e6f9f4d18\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7f964fb678lhs8d" Jul 13 22:18:56.224829 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:18:56.224807 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/318be4cb-7ac5-4039-a1be-f26e6f9f4d18-istio-token\") pod \"data-science-gateway-data-science-gateway-class-7f964fb678lhs8d\" (UID: \"318be4cb-7ac5-4039-a1be-f26e6f9f4d18\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7f964fb678lhs8d" Jul 13 22:18:56.225195 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:18:56.224837 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/318be4cb-7ac5-4039-a1be-f26e6f9f4d18-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-7f964fb678lhs8d\" (UID: \"318be4cb-7ac5-4039-a1be-f26e6f9f4d18\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7f964fb678lhs8d" Jul 13 22:18:56.225195 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:18:56.225077 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/318be4cb-7ac5-4039-a1be-f26e6f9f4d18-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-7f964fb678lhs8d\" (UID: \"318be4cb-7ac5-4039-a1be-f26e6f9f4d18\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7f964fb678lhs8d" Jul 13 22:18:56.225195 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:18:56.225159 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/318be4cb-7ac5-4039-a1be-f26e6f9f4d18-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-7f964fb678lhs8d\" (UID: \"318be4cb-7ac5-4039-a1be-f26e6f9f4d18\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7f964fb678lhs8d" Jul 13 22:18:56.225344 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:18:56.225214 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/318be4cb-7ac5-4039-a1be-f26e6f9f4d18-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-7f964fb678lhs8d\" (UID: \"318be4cb-7ac5-4039-a1be-f26e6f9f4d18\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7f964fb678lhs8d" Jul 13 22:18:56.225410 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:18:56.225367 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/318be4cb-7ac5-4039-a1be-f26e6f9f4d18-istio-data\") pod \"data-science-gateway-data-science-gateway-class-7f964fb678lhs8d\" (UID: \"318be4cb-7ac5-4039-a1be-f26e6f9f4d18\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7f964fb678lhs8d" Jul 13 22:18:56.225590 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:18:56.225545 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/318be4cb-7ac5-4039-a1be-f26e6f9f4d18-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-7f964fb678lhs8d\" (UID: \"318be4cb-7ac5-4039-a1be-f26e6f9f4d18\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7f964fb678lhs8d" Jul 13 22:18:56.227137 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:18:56.227118 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/318be4cb-7ac5-4039-a1be-f26e6f9f4d18-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-7f964fb678lhs8d\" (UID: \"318be4cb-7ac5-4039-a1be-f26e6f9f4d18\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7f964fb678lhs8d" Jul 13 22:18:56.227608 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:18:56.227591 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/318be4cb-7ac5-4039-a1be-f26e6f9f4d18-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-7f964fb678lhs8d\" (UID: \"318be4cb-7ac5-4039-a1be-f26e6f9f4d18\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7f964fb678lhs8d" Jul 13 22:18:56.241895 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:18:56.241865 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gb5xh\" (UniqueName: \"kubernetes.io/projected/318be4cb-7ac5-4039-a1be-f26e6f9f4d18-kube-api-access-gb5xh\") pod \"data-science-gateway-data-science-gateway-class-7f964fb678lhs8d\" (UID: \"318be4cb-7ac5-4039-a1be-f26e6f9f4d18\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7f964fb678lhs8d" Jul 13 22:18:56.245236 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:18:56.245215 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/318be4cb-7ac5-4039-a1be-f26e6f9f4d18-istio-token\") pod \"data-science-gateway-data-science-gateway-class-7f964fb678lhs8d\" (UID: \"318be4cb-7ac5-4039-a1be-f26e6f9f4d18\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7f964fb678lhs8d" Jul 13 22:18:56.421817 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:18:56.421720 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7f964fb678lhs8d" Jul 13 22:18:56.556878 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:18:56.556840 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-7f964fb678lhs8d"] Jul 13 22:18:56.559790 ip-10-0-133-179 kubenswrapper[2583]: W0713 22:18:56.559762 2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod318be4cb_7ac5_4039_a1be_f26e6f9f4d18.slice/crio-a27bd3ce54b87f231e826e26d2118c387d63534cd86d787eb42544e07208fcbc WatchSource:0}: Error finding container a27bd3ce54b87f231e826e26d2118c387d63534cd86d787eb42544e07208fcbc: Status 404 returned error can't find the container with id a27bd3ce54b87f231e826e26d2118c387d63534cd86d787eb42544e07208fcbc Jul 13 22:18:56.644755 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:18:56.644719 2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-94bb8fbfd-28vcl"] Jul 13 22:18:56.647786 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:18:56.647769 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-28vcl" Jul 13 22:18:56.650659 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:18:56.650636 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istio-kubeconfig\"" Jul 13 22:18:56.650797 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:18:56.650636 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-openshift-gateway-dockercfg-2kxsc\"" Jul 13 22:18:56.650797 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:18:56.650694 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"cacerts\"" Jul 13 22:18:56.650797 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:18:56.650775 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-tls\"" Jul 13 22:18:56.661529 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:18:56.661505 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-94bb8fbfd-28vcl"] Jul 13 22:18:56.689760 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:18:56.689648 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7f964fb678lhs8d" event={"ID":"318be4cb-7ac5-4039-a1be-f26e6f9f4d18","Type":"ContainerStarted","Data":"a27bd3ce54b87f231e826e26d2118c387d63534cd86d787eb42544e07208fcbc"} Jul 13 22:18:56.728266 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:18:56.728223 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/52bdd097-54ac-4137-aced-c5b4debbb854-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-94bb8fbfd-28vcl\" (UID: \"52bdd097-54ac-4137-aced-c5b4debbb854\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-28vcl" Jul 13 22:18:56.728447 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:18:56.728280 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/52bdd097-54ac-4137-aced-c5b4debbb854-istio-kubeconfig\") pod \"istiod-openshift-gateway-94bb8fbfd-28vcl\" (UID: \"52bdd097-54ac-4137-aced-c5b4debbb854\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-28vcl" Jul 13 22:18:56.728447 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:18:56.728304 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8dtcb\" (UniqueName: \"kubernetes.io/projected/52bdd097-54ac-4137-aced-c5b4debbb854-kube-api-access-8dtcb\") pod \"istiod-openshift-gateway-94bb8fbfd-28vcl\" (UID: \"52bdd097-54ac-4137-aced-c5b4debbb854\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-28vcl" Jul 13 22:18:56.728447 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:18:56.728366 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/52bdd097-54ac-4137-aced-c5b4debbb854-cacerts\") pod \"istiod-openshift-gateway-94bb8fbfd-28vcl\" (UID: \"52bdd097-54ac-4137-aced-c5b4debbb854\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-28vcl" Jul 13 22:18:56.728447 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:18:56.728390 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/52bdd097-54ac-4137-aced-c5b4debbb854-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-94bb8fbfd-28vcl\" (UID: \"52bdd097-54ac-4137-aced-c5b4debbb854\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-28vcl" Jul 13 22:18:56.728447 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:18:56.728428 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/52bdd097-54ac-4137-aced-c5b4debbb854-local-certs\") pod \"istiod-openshift-gateway-94bb8fbfd-28vcl\" (UID: \"52bdd097-54ac-4137-aced-c5b4debbb854\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-28vcl" Jul 13 22:18:56.728652 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:18:56.728449 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/52bdd097-54ac-4137-aced-c5b4debbb854-istio-token\") pod \"istiod-openshift-gateway-94bb8fbfd-28vcl\" (UID: \"52bdd097-54ac-4137-aced-c5b4debbb854\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-28vcl" Jul 13 22:18:56.828940 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:18:56.828892 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/52bdd097-54ac-4137-aced-c5b4debbb854-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-94bb8fbfd-28vcl\" (UID: \"52bdd097-54ac-4137-aced-c5b4debbb854\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-28vcl" Jul 13 22:18:56.829145 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:18:56.828959 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/52bdd097-54ac-4137-aced-c5b4debbb854-istio-kubeconfig\") pod \"istiod-openshift-gateway-94bb8fbfd-28vcl\" (UID: \"52bdd097-54ac-4137-aced-c5b4debbb854\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-28vcl" Jul 13 22:18:56.829145 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:18:56.828980 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8dtcb\" (UniqueName: \"kubernetes.io/projected/52bdd097-54ac-4137-aced-c5b4debbb854-kube-api-access-8dtcb\") pod \"istiod-openshift-gateway-94bb8fbfd-28vcl\" (UID: \"52bdd097-54ac-4137-aced-c5b4debbb854\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-28vcl" Jul 13 22:18:56.829145 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:18:56.829021 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/52bdd097-54ac-4137-aced-c5b4debbb854-cacerts\") pod \"istiod-openshift-gateway-94bb8fbfd-28vcl\" (UID: \"52bdd097-54ac-4137-aced-c5b4debbb854\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-28vcl" Jul 13 22:18:56.829145 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:18:56.829045 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/52bdd097-54ac-4137-aced-c5b4debbb854-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-94bb8fbfd-28vcl\" (UID: \"52bdd097-54ac-4137-aced-c5b4debbb854\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-28vcl" Jul 13 22:18:56.829358 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:18:56.829208 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/52bdd097-54ac-4137-aced-c5b4debbb854-local-certs\") pod \"istiod-openshift-gateway-94bb8fbfd-28vcl\" (UID: \"52bdd097-54ac-4137-aced-c5b4debbb854\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-28vcl" Jul 13 22:18:56.829358 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:18:56.829247 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/52bdd097-54ac-4137-aced-c5b4debbb854-istio-token\") pod \"istiod-openshift-gateway-94bb8fbfd-28vcl\" (UID: \"52bdd097-54ac-4137-aced-c5b4debbb854\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-28vcl" Jul 13 22:18:56.829766 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:18:56.829743 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/52bdd097-54ac-4137-aced-c5b4debbb854-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-94bb8fbfd-28vcl\" (UID: \"52bdd097-54ac-4137-aced-c5b4debbb854\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-28vcl" Jul 13 22:18:56.831327 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:18:56.831298 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/52bdd097-54ac-4137-aced-c5b4debbb854-local-certs\") pod \"istiod-openshift-gateway-94bb8fbfd-28vcl\" (UID: \"52bdd097-54ac-4137-aced-c5b4debbb854\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-28vcl" Jul 13 22:18:56.831435 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:18:56.831392 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/52bdd097-54ac-4137-aced-c5b4debbb854-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-94bb8fbfd-28vcl\" (UID: \"52bdd097-54ac-4137-aced-c5b4debbb854\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-28vcl" Jul 13 22:18:56.831519 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:18:56.831499 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/52bdd097-54ac-4137-aced-c5b4debbb854-cacerts\") pod \"istiod-openshift-gateway-94bb8fbfd-28vcl\" (UID: \"52bdd097-54ac-4137-aced-c5b4debbb854\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-28vcl" Jul 13 22:18:56.831595 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:18:56.831515 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/52bdd097-54ac-4137-aced-c5b4debbb854-istio-kubeconfig\") pod \"istiod-openshift-gateway-94bb8fbfd-28vcl\" (UID: \"52bdd097-54ac-4137-aced-c5b4debbb854\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-28vcl" Jul 13 22:18:56.856875 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:18:56.856848 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8dtcb\" (UniqueName: \"kubernetes.io/projected/52bdd097-54ac-4137-aced-c5b4debbb854-kube-api-access-8dtcb\") pod \"istiod-openshift-gateway-94bb8fbfd-28vcl\" (UID: \"52bdd097-54ac-4137-aced-c5b4debbb854\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-28vcl" Jul 13 22:18:56.857191 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:18:56.857171 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/52bdd097-54ac-4137-aced-c5b4debbb854-istio-token\") pod \"istiod-openshift-gateway-94bb8fbfd-28vcl\" (UID: \"52bdd097-54ac-4137-aced-c5b4debbb854\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-28vcl" Jul 13 22:18:56.957062 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:18:56.956982 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-28vcl" Jul 13 22:18:57.104468 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:18:57.104446 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-94bb8fbfd-28vcl"] Jul 13 22:18:57.105671 ip-10-0-133-179 kubenswrapper[2583]: W0713 22:18:57.105638 2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod52bdd097_54ac_4137_aced_c5b4debbb854.slice/crio-2888478c384de00c4ecfe5b889be0ec432fc4bdcaef33f69ea72bc6dc584d4b8 WatchSource:0}: Error finding container 2888478c384de00c4ecfe5b889be0ec432fc4bdcaef33f69ea72bc6dc584d4b8: Status 404 returned error can't find the container with id 2888478c384de00c4ecfe5b889be0ec432fc4bdcaef33f69ea72bc6dc584d4b8 Jul 13 22:18:57.695901 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:18:57.695855 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-28vcl" event={"ID":"52bdd097-54ac-4137-aced-c5b4debbb854","Type":"ContainerStarted","Data":"2888478c384de00c4ecfe5b889be0ec432fc4bdcaef33f69ea72bc6dc584d4b8"} Jul 13 22:19:00.385200 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:19:00.385163 2583 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 13 22:19:00.385541 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:19:00.385249 2583 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 13 22:19:00.385541 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:19:00.385297 2583 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 13 22:19:00.385541 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:19:00.385434 2583 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 13 22:19:00.385541 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:19:00.385486 2583 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 13 22:19:00.710644 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:19:00.710540 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-28vcl" event={"ID":"52bdd097-54ac-4137-aced-c5b4debbb854","Type":"ContainerStarted","Data":"4357231f29f05df7fa9b1f8dada87516837839fe9cf77f1eb83d88fe0be89afe"} Jul 13 22:19:00.710800 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:19:00.710704 2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-28vcl" Jul 13 22:19:00.712891 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:19:00.712324 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7f964fb678lhs8d" event={"ID":"318be4cb-7ac5-4039-a1be-f26e6f9f4d18","Type":"ContainerStarted","Data":"2d88b6e315fa6dad5d0bd0e4788f5415342bc8ecb00774cc1dde90f5b76ebaf2"} Jul 13 22:19:00.749758 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:19:00.749683 2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-28vcl" podStartSLOduration=1.472003359 podStartE2EDuration="4.749661874s" totalImagesPullingTime="3.277658515s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-13 22:18:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 22:19:00.746930108 +0000 UTC m=+335.994845459" watchObservedRunningTime="2026-07-13 22:19:00.749661874 +0000 UTC m=+335.997577225" Jul 13 22:19:00.783233 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:19:00.783165 2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7f964fb678lhs8d" podStartSLOduration=0.959892021 podStartE2EDuration="4.783143481s" totalImagesPullingTime="3.82325146s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-13 22:18:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 22:19:00.781180892 +0000 UTC m=+336.029096243" watchObservedRunningTime="2026-07-13 22:19:00.783143481 +0000 UTC m=+336.031058833" Jul 13 22:19:01.422332 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:19:01.422284 2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7f964fb678lhs8d" Jul 13 22:19:01.427047 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:19:01.427022 2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7f964fb678lhs8d" Jul 13 22:19:01.716636 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:19:01.716533 2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7f964fb678lhs8d" Jul 13 22:19:01.717393 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:19:01.717367 2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7f964fb678lhs8d" Jul 13 22:19:01.718135 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:19:01.718112 2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-28vcl" Jul 13 22:22:54.832794 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:22:54.832699 2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-5c664664b-b9ccj"] Jul 13 22:22:54.834778 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:22:54.834759 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-5c664664b-b9ccj" Jul 13 22:22:54.836853 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:22:54.836827 2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Jul 13 22:22:54.836978 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:22:54.836833 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-tgdl9\"" Jul 13 22:22:54.837511 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:22:54.837495 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-serving-cert\"" Jul 13 22:22:54.837626 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:22:54.837530 2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Jul 13 22:22:54.844391 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:22:54.844366 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-5c664664b-b9ccj"] Jul 13 22:22:54.888987 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:22:54.888952 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c2e249b2-a08e-44ca-a911-52799b788c09-maas-api-tls\") pod \"maas-api-5c664664b-b9ccj\" (UID: \"c2e249b2-a08e-44ca-a911-52799b788c09\") " pod="odh-ai-gateway-infra/maas-api-5c664664b-b9ccj" Jul 13 22:22:54.888987 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:22:54.888987 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-khkr6\" (UniqueName: \"kubernetes.io/projected/c2e249b2-a08e-44ca-a911-52799b788c09-kube-api-access-khkr6\") pod \"maas-api-5c664664b-b9ccj\" (UID: \"c2e249b2-a08e-44ca-a911-52799b788c09\") " pod="odh-ai-gateway-infra/maas-api-5c664664b-b9ccj" Jul 13 22:22:54.889228 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:22:54.889010 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c2e249b2-a08e-44ca-a911-52799b788c09-openshift-service-ca\") pod \"maas-api-5c664664b-b9ccj\" (UID: \"c2e249b2-a08e-44ca-a911-52799b788c09\") " pod="odh-ai-gateway-infra/maas-api-5c664664b-b9ccj" Jul 13 22:22:54.989906 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:22:54.989855 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c2e249b2-a08e-44ca-a911-52799b788c09-maas-api-tls\") pod \"maas-api-5c664664b-b9ccj\" (UID: \"c2e249b2-a08e-44ca-a911-52799b788c09\") " pod="odh-ai-gateway-infra/maas-api-5c664664b-b9ccj" Jul 13 22:22:54.989906 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:22:54.989911 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-khkr6\" (UniqueName: \"kubernetes.io/projected/c2e249b2-a08e-44ca-a911-52799b788c09-kube-api-access-khkr6\") pod \"maas-api-5c664664b-b9ccj\" (UID: \"c2e249b2-a08e-44ca-a911-52799b788c09\") " pod="odh-ai-gateway-infra/maas-api-5c664664b-b9ccj" Jul 13 22:22:54.990109 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:22:54.989940 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c2e249b2-a08e-44ca-a911-52799b788c09-openshift-service-ca\") pod \"maas-api-5c664664b-b9ccj\" (UID: \"c2e249b2-a08e-44ca-a911-52799b788c09\") " pod="odh-ai-gateway-infra/maas-api-5c664664b-b9ccj" Jul 13 22:22:54.990655 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:22:54.990632 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c2e249b2-a08e-44ca-a911-52799b788c09-openshift-service-ca\") pod \"maas-api-5c664664b-b9ccj\" (UID: \"c2e249b2-a08e-44ca-a911-52799b788c09\") " pod="odh-ai-gateway-infra/maas-api-5c664664b-b9ccj" Jul 13 22:22:54.992318 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:22:54.992297 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c2e249b2-a08e-44ca-a911-52799b788c09-maas-api-tls\") pod \"maas-api-5c664664b-b9ccj\" (UID: \"c2e249b2-a08e-44ca-a911-52799b788c09\") " pod="odh-ai-gateway-infra/maas-api-5c664664b-b9ccj" Jul 13 22:22:54.997987 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:22:54.997960 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-khkr6\" (UniqueName: \"kubernetes.io/projected/c2e249b2-a08e-44ca-a911-52799b788c09-kube-api-access-khkr6\") pod \"maas-api-5c664664b-b9ccj\" (UID: \"c2e249b2-a08e-44ca-a911-52799b788c09\") " pod="odh-ai-gateway-infra/maas-api-5c664664b-b9ccj" Jul 13 22:22:55.145217 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:22:55.145130 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-5c664664b-b9ccj" Jul 13 22:22:55.276723 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:22:55.276691 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-5c664664b-b9ccj"] Jul 13 22:22:55.281552 ip-10-0-133-179 kubenswrapper[2583]: W0713 22:22:55.281524 2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc2e249b2_a08e_44ca_a911_52799b788c09.slice/crio-6119731b3bbadba5e20dedd54e546abd5b7237f5dc2af67c8cd8ef9186944a99 WatchSource:0}: Error finding container 6119731b3bbadba5e20dedd54e546abd5b7237f5dc2af67c8cd8ef9186944a99: Status 404 returned error can't find the container with id 6119731b3bbadba5e20dedd54e546abd5b7237f5dc2af67c8cd8ef9186944a99 Jul 13 22:22:55.379098 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:22:55.379065 2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-7964457696-49fdw"] Jul 13 22:22:55.381971 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:22:55.381947 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-7964457696-49fdw" Jul 13 22:22:55.384822 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:22:55.384799 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-dockercfg-lv25x\"" Jul 13 22:22:55.384942 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:22:55.384799 2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins\"" Jul 13 22:22:55.391816 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:22:55.391790 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-7964457696-49fdw"] Jul 13 22:22:55.493691 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:22:55.493596 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/617e7821-66c5-4515-b29a-b0e3ed27655f-plugins-config-volume\") pod \"payload-pre-processing-7964457696-49fdw\" (UID: \"617e7821-66c5-4515-b29a-b0e3ed27655f\") " pod="openshift-ingress/payload-pre-processing-7964457696-49fdw" Jul 13 22:22:55.493834 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:22:55.493720 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8nrnw\" (UniqueName: \"kubernetes.io/projected/617e7821-66c5-4515-b29a-b0e3ed27655f-kube-api-access-8nrnw\") pod \"payload-pre-processing-7964457696-49fdw\" (UID: \"617e7821-66c5-4515-b29a-b0e3ed27655f\") " pod="openshift-ingress/payload-pre-processing-7964457696-49fdw" Jul 13 22:22:55.544110 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:22:55.544069 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-5c664664b-b9ccj" event={"ID":"c2e249b2-a08e-44ca-a911-52799b788c09","Type":"ContainerStarted","Data":"6119731b3bbadba5e20dedd54e546abd5b7237f5dc2af67c8cd8ef9186944a99"} Jul 13 22:22:55.594556 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:22:55.594513 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8nrnw\" (UniqueName: \"kubernetes.io/projected/617e7821-66c5-4515-b29a-b0e3ed27655f-kube-api-access-8nrnw\") pod \"payload-pre-processing-7964457696-49fdw\" (UID: \"617e7821-66c5-4515-b29a-b0e3ed27655f\") " pod="openshift-ingress/payload-pre-processing-7964457696-49fdw" Jul 13 22:22:55.594556 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:22:55.594561 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/617e7821-66c5-4515-b29a-b0e3ed27655f-plugins-config-volume\") pod \"payload-pre-processing-7964457696-49fdw\" (UID: \"617e7821-66c5-4515-b29a-b0e3ed27655f\") " pod="openshift-ingress/payload-pre-processing-7964457696-49fdw" Jul 13 22:22:55.595316 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:22:55.595286 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/617e7821-66c5-4515-b29a-b0e3ed27655f-plugins-config-volume\") pod \"payload-pre-processing-7964457696-49fdw\" (UID: \"617e7821-66c5-4515-b29a-b0e3ed27655f\") " pod="openshift-ingress/payload-pre-processing-7964457696-49fdw" Jul 13 22:22:55.602109 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:22:55.602077 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8nrnw\" (UniqueName: \"kubernetes.io/projected/617e7821-66c5-4515-b29a-b0e3ed27655f-kube-api-access-8nrnw\") pod \"payload-pre-processing-7964457696-49fdw\" (UID: \"617e7821-66c5-4515-b29a-b0e3ed27655f\") " pod="openshift-ingress/payload-pre-processing-7964457696-49fdw" Jul 13 22:22:55.694065 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:22:55.694021 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-7964457696-49fdw" Jul 13 22:22:55.840561 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:22:55.840528 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-7964457696-49fdw"] Jul 13 22:22:55.844002 ip-10-0-133-179 kubenswrapper[2583]: W0713 22:22:55.843959 2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod617e7821_66c5_4515_b29a_b0e3ed27655f.slice/crio-db8ad223d3fe4afbd65ec85a2751a05173e7c8a7e387f66c60fc0b905414b734 WatchSource:0}: Error finding container db8ad223d3fe4afbd65ec85a2751a05173e7c8a7e387f66c60fc0b905414b734: Status 404 returned error can't find the container with id db8ad223d3fe4afbd65ec85a2751a05173e7c8a7e387f66c60fc0b905414b734 Jul 13 22:22:56.550101 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:22:56.550057 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-7964457696-49fdw" event={"ID":"617e7821-66c5-4515-b29a-b0e3ed27655f","Type":"ContainerStarted","Data":"db8ad223d3fe4afbd65ec85a2751a05173e7c8a7e387f66c60fc0b905414b734"} Jul 13 22:22:57.557150 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:22:57.557115 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-5c664664b-b9ccj" event={"ID":"c2e249b2-a08e-44ca-a911-52799b788c09","Type":"ContainerStarted","Data":"46e3c7372b5475ba1c12020a620b8b98277beff554882ea86ab573b146e657f7"} Jul 13 22:22:57.557728 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:22:57.557244 2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-5c664664b-b9ccj" Jul 13 22:22:57.573454 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:22:57.573392 2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-5c664664b-b9ccj" podStartSLOduration=1.448630234 podStartE2EDuration="3.573368617s" totalImagesPullingTime="2.124738383s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-13 22:22:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 22:22:57.571466138 +0000 UTC m=+572.819381488" watchObservedRunningTime="2026-07-13 22:22:57.573368617 +0000 UTC m=+572.821283968" Jul 13 22:22:58.562322 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:22:58.562288 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-7964457696-49fdw" event={"ID":"617e7821-66c5-4515-b29a-b0e3ed27655f","Type":"ContainerStarted","Data":"f366a890ea8fe9e89ef37d876ffaa1464ee307e591b80301831b980335e2194c"} Jul 13 22:22:58.562719 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:22:58.562348 2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-7964457696-49fdw" Jul 13 22:22:58.581134 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:22:58.581074 2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-7964457696-49fdw" podStartSLOduration=1.306245023 podStartE2EDuration="3.581054417s" totalImagesPullingTime="2.274809394s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-13 22:22:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 22:22:58.580906146 +0000 UTC m=+573.828821522" watchObservedRunningTime="2026-07-13 22:22:58.581054417 +0000 UTC m=+573.828969769" Jul 13 22:23:03.568400 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:23:03.568373 2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-5c664664b-b9ccj" Jul 13 22:23:09.567509 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:23:09.567479 2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-7964457696-49fdw" Jul 13 22:23:19.271062 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:23:19.271028 2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-distinct-simulated-kserve-59975d69f9-5jd66"] Jul 13 22:23:19.274028 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:23:19.274009 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-5jd66" Jul 13 22:23:19.276206 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:23:19.276151 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-2zd2s\"" Jul 13 22:23:19.276206 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:23:19.276183 2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\"" Jul 13 22:23:19.276971 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:23:19.276949 2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\"" Jul 13 22:23:19.277069 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:23:19.277000 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-distinct-simulated-kserve-self-signed-certs\"" Jul 13 22:23:19.282328 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:23:19.282306 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-simulated-kserve-59975d69f9-5jd66"] Jul 13 22:23:19.391735 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:23:19.391705 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/81dac3d5-64d0-4c88-9350-07e5b0fc18fc-model-cache\") pod \"e2e-distinct-simulated-kserve-59975d69f9-5jd66\" (UID: \"81dac3d5-64d0-4c88-9350-07e5b0fc18fc\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-5jd66" Jul 13 22:23:19.391891 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:23:19.391743 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/81dac3d5-64d0-4c88-9350-07e5b0fc18fc-home\") pod \"e2e-distinct-simulated-kserve-59975d69f9-5jd66\" (UID: \"81dac3d5-64d0-4c88-9350-07e5b0fc18fc\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-5jd66" Jul 13 22:23:19.391891 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:23:19.391761 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/81dac3d5-64d0-4c88-9350-07e5b0fc18fc-tmp-dir\") pod \"e2e-distinct-simulated-kserve-59975d69f9-5jd66\" (UID: \"81dac3d5-64d0-4c88-9350-07e5b0fc18fc\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-5jd66" Jul 13 22:23:19.391891 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:23:19.391780 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/81dac3d5-64d0-4c88-9350-07e5b0fc18fc-dshm\") pod \"e2e-distinct-simulated-kserve-59975d69f9-5jd66\" (UID: \"81dac3d5-64d0-4c88-9350-07e5b0fc18fc\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-5jd66" Jul 13 22:23:19.391891 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:23:19.391845 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/81dac3d5-64d0-4c88-9350-07e5b0fc18fc-tls-certs\") pod \"e2e-distinct-simulated-kserve-59975d69f9-5jd66\" (UID: \"81dac3d5-64d0-4c88-9350-07e5b0fc18fc\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-5jd66" Jul 13 22:23:19.392038 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:23:19.391897 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-frmvr\" (UniqueName: \"kubernetes.io/projected/81dac3d5-64d0-4c88-9350-07e5b0fc18fc-kube-api-access-frmvr\") pod \"e2e-distinct-simulated-kserve-59975d69f9-5jd66\" (UID: \"81dac3d5-64d0-4c88-9350-07e5b0fc18fc\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-5jd66" Jul 13 22:23:19.492881 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:23:19.492836 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/81dac3d5-64d0-4c88-9350-07e5b0fc18fc-model-cache\") pod \"e2e-distinct-simulated-kserve-59975d69f9-5jd66\" (UID: \"81dac3d5-64d0-4c88-9350-07e5b0fc18fc\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-5jd66" Jul 13 22:23:19.492881 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:23:19.492885 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/81dac3d5-64d0-4c88-9350-07e5b0fc18fc-home\") pod \"e2e-distinct-simulated-kserve-59975d69f9-5jd66\" (UID: \"81dac3d5-64d0-4c88-9350-07e5b0fc18fc\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-5jd66" Jul 13 22:23:19.493127 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:23:19.492901 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/81dac3d5-64d0-4c88-9350-07e5b0fc18fc-tmp-dir\") pod \"e2e-distinct-simulated-kserve-59975d69f9-5jd66\" (UID: \"81dac3d5-64d0-4c88-9350-07e5b0fc18fc\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-5jd66" Jul 13 22:23:19.493127 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:23:19.492920 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/81dac3d5-64d0-4c88-9350-07e5b0fc18fc-dshm\") pod \"e2e-distinct-simulated-kserve-59975d69f9-5jd66\" (UID: \"81dac3d5-64d0-4c88-9350-07e5b0fc18fc\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-5jd66" Jul 13 22:23:19.493127 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:23:19.492944 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/81dac3d5-64d0-4c88-9350-07e5b0fc18fc-tls-certs\") pod \"e2e-distinct-simulated-kserve-59975d69f9-5jd66\" (UID: \"81dac3d5-64d0-4c88-9350-07e5b0fc18fc\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-5jd66" Jul 13 22:23:19.493127 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:23:19.492970 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-frmvr\" (UniqueName: \"kubernetes.io/projected/81dac3d5-64d0-4c88-9350-07e5b0fc18fc-kube-api-access-frmvr\") pod \"e2e-distinct-simulated-kserve-59975d69f9-5jd66\" (UID: \"81dac3d5-64d0-4c88-9350-07e5b0fc18fc\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-5jd66" Jul 13 22:23:19.493387 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:23:19.493362 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/81dac3d5-64d0-4c88-9350-07e5b0fc18fc-home\") pod \"e2e-distinct-simulated-kserve-59975d69f9-5jd66\" (UID: \"81dac3d5-64d0-4c88-9350-07e5b0fc18fc\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-5jd66" Jul 13 22:23:19.493387 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:23:19.493375 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/81dac3d5-64d0-4c88-9350-07e5b0fc18fc-model-cache\") pod \"e2e-distinct-simulated-kserve-59975d69f9-5jd66\" (UID: \"81dac3d5-64d0-4c88-9350-07e5b0fc18fc\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-5jd66" Jul 13 22:23:19.493502 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:23:19.493400 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/81dac3d5-64d0-4c88-9350-07e5b0fc18fc-tmp-dir\") pod \"e2e-distinct-simulated-kserve-59975d69f9-5jd66\" (UID: \"81dac3d5-64d0-4c88-9350-07e5b0fc18fc\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-5jd66" Jul 13 22:23:19.495251 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:23:19.495226 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/81dac3d5-64d0-4c88-9350-07e5b0fc18fc-dshm\") pod \"e2e-distinct-simulated-kserve-59975d69f9-5jd66\" (UID: \"81dac3d5-64d0-4c88-9350-07e5b0fc18fc\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-5jd66" Jul 13 22:23:19.495476 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:23:19.495458 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/81dac3d5-64d0-4c88-9350-07e5b0fc18fc-tls-certs\") pod \"e2e-distinct-simulated-kserve-59975d69f9-5jd66\" (UID: \"81dac3d5-64d0-4c88-9350-07e5b0fc18fc\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-5jd66" Jul 13 22:23:19.500793 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:23:19.500775 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-frmvr\" (UniqueName: \"kubernetes.io/projected/81dac3d5-64d0-4c88-9350-07e5b0fc18fc-kube-api-access-frmvr\") pod \"e2e-distinct-simulated-kserve-59975d69f9-5jd66\" (UID: \"81dac3d5-64d0-4c88-9350-07e5b0fc18fc\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-5jd66" Jul 13 22:23:19.585516 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:23:19.585426 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-5jd66" Jul 13 22:23:19.712806 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:23:19.712774 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-simulated-kserve-59975d69f9-5jd66"] Jul 13 22:23:19.715780 ip-10-0-133-179 kubenswrapper[2583]: W0713 22:23:19.715750 2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod81dac3d5_64d0_4c88_9350_07e5b0fc18fc.slice/crio-b75039ce921283f8c99af344415af91c36de715ca91a231846e27690ec618e38 WatchSource:0}: Error finding container b75039ce921283f8c99af344415af91c36de715ca91a231846e27690ec618e38: Status 404 returned error can't find the container with id b75039ce921283f8c99af344415af91c36de715ca91a231846e27690ec618e38 Jul 13 22:23:20.651636 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:23:20.651596 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-5jd66" event={"ID":"81dac3d5-64d0-4c88-9350-07e5b0fc18fc","Type":"ContainerStarted","Data":"b75039ce921283f8c99af344415af91c36de715ca91a231846e27690ec618e38"} Jul 13 22:23:21.657366 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:23:21.657327 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-5jd66" event={"ID":"81dac3d5-64d0-4c88-9350-07e5b0fc18fc","Type":"ContainerStarted","Data":"0a51cb28a2e19f4840aff5395eaaaed5e812bb949b4da7298f070f514caba5a3"} Jul 13 22:23:21.673677 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:23:21.673625 2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-5jd66" podStartSLOduration=1.664702255 podStartE2EDuration="2.673609321s" totalImagesPullingTime="1.008907066s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-13 22:23:19 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 22:23:21.672345143 +0000 UTC m=+596.920260493" watchObservedRunningTime="2026-07-13 22:23:21.673609321 +0000 UTC m=+596.921524670" Jul 13 22:23:29.585962 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:23:29.585924 2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-5jd66" Jul 13 22:23:29.591644 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:23:29.585992 2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-5jd66" Jul 13 22:23:29.598660 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:23:29.598632 2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-5jd66" Jul 13 22:23:29.698744 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:23:29.698710 2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-5jd66" Jul 13 22:23:34.872312 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:23:34.872280 2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-74b68447df-l8v2f"] Jul 13 22:23:34.877238 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:23:34.877215 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-l8v2f" Jul 13 22:23:34.879610 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:23:34.879550 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"facebook-opt-125m-simulated-kserve-self-signed-certs\"" Jul 13 22:23:34.885659 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:23:34.885638 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-74b68447df-l8v2f"] Jul 13 22:23:34.921071 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:23:34.921039 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bf682d14-b0e6-484f-bad2-ac5790938edf-dshm\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-l8v2f\" (UID: \"bf682d14-b0e6-484f-bad2-ac5790938edf\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-l8v2f" Jul 13 22:23:34.921237 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:23:34.921085 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bf682d14-b0e6-484f-bad2-ac5790938edf-home\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-l8v2f\" (UID: \"bf682d14-b0e6-484f-bad2-ac5790938edf\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-l8v2f" Jul 13 22:23:34.921237 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:23:34.921134 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bf682d14-b0e6-484f-bad2-ac5790938edf-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-l8v2f\" (UID: \"bf682d14-b0e6-484f-bad2-ac5790938edf\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-l8v2f" Jul 13 22:23:34.921237 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:23:34.921184 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bf682d14-b0e6-484f-bad2-ac5790938edf-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-l8v2f\" (UID: \"bf682d14-b0e6-484f-bad2-ac5790938edf\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-l8v2f" Jul 13 22:23:34.921237 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:23:34.921223 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bf682d14-b0e6-484f-bad2-ac5790938edf-model-cache\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-l8v2f\" (UID: \"bf682d14-b0e6-484f-bad2-ac5790938edf\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-l8v2f" Jul 13 22:23:34.921396 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:23:34.921252 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hff5h\" (UniqueName: \"kubernetes.io/projected/bf682d14-b0e6-484f-bad2-ac5790938edf-kube-api-access-hff5h\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-l8v2f\" (UID: \"bf682d14-b0e6-484f-bad2-ac5790938edf\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-l8v2f" Jul 13 22:23:35.021654 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:23:35.021617 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bf682d14-b0e6-484f-bad2-ac5790938edf-model-cache\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-l8v2f\" (UID: \"bf682d14-b0e6-484f-bad2-ac5790938edf\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-l8v2f" Jul 13 22:23:35.021654 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:23:35.021659 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hff5h\" (UniqueName: \"kubernetes.io/projected/bf682d14-b0e6-484f-bad2-ac5790938edf-kube-api-access-hff5h\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-l8v2f\" (UID: \"bf682d14-b0e6-484f-bad2-ac5790938edf\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-l8v2f" Jul 13 22:23:35.021872 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:23:35.021693 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bf682d14-b0e6-484f-bad2-ac5790938edf-dshm\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-l8v2f\" (UID: \"bf682d14-b0e6-484f-bad2-ac5790938edf\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-l8v2f" Jul 13 22:23:35.021872 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:23:35.021728 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bf682d14-b0e6-484f-bad2-ac5790938edf-home\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-l8v2f\" (UID: \"bf682d14-b0e6-484f-bad2-ac5790938edf\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-l8v2f" Jul 13 22:23:35.021872 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:23:35.021750 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bf682d14-b0e6-484f-bad2-ac5790938edf-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-l8v2f\" (UID: \"bf682d14-b0e6-484f-bad2-ac5790938edf\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-l8v2f" Jul 13 22:23:35.022010 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:23:35.021920 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bf682d14-b0e6-484f-bad2-ac5790938edf-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-l8v2f\" (UID: \"bf682d14-b0e6-484f-bad2-ac5790938edf\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-l8v2f" Jul 13 22:23:35.022068 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:23:35.022048 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bf682d14-b0e6-484f-bad2-ac5790938edf-model-cache\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-l8v2f\" (UID: \"bf682d14-b0e6-484f-bad2-ac5790938edf\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-l8v2f" Jul 13 22:23:35.022119 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:23:35.022081 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bf682d14-b0e6-484f-bad2-ac5790938edf-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-l8v2f\" (UID: \"bf682d14-b0e6-484f-bad2-ac5790938edf\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-l8v2f" Jul 13 22:23:35.022171 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:23:35.022144 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bf682d14-b0e6-484f-bad2-ac5790938edf-home\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-l8v2f\" (UID: \"bf682d14-b0e6-484f-bad2-ac5790938edf\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-l8v2f" Jul 13 22:23:35.024045 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:23:35.024019 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bf682d14-b0e6-484f-bad2-ac5790938edf-dshm\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-l8v2f\" (UID: \"bf682d14-b0e6-484f-bad2-ac5790938edf\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-l8v2f" Jul 13 22:23:35.024262 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:23:35.024243 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bf682d14-b0e6-484f-bad2-ac5790938edf-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-l8v2f\" (UID: \"bf682d14-b0e6-484f-bad2-ac5790938edf\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-l8v2f" Jul 13 22:23:35.029971 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:23:35.029947 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hff5h\" (UniqueName: \"kubernetes.io/projected/bf682d14-b0e6-484f-bad2-ac5790938edf-kube-api-access-hff5h\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-l8v2f\" (UID: \"bf682d14-b0e6-484f-bad2-ac5790938edf\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-l8v2f" Jul 13 22:23:35.189994 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:23:35.189902 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-l8v2f" Jul 13 22:23:35.313404 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:23:35.313345 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-74b68447df-l8v2f"] Jul 13 22:23:35.315922 ip-10-0-133-179 kubenswrapper[2583]: W0713 22:23:35.315897 2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbf682d14_b0e6_484f_bad2_ac5790938edf.slice/crio-809b9cd8a8cf9c692a9d5ea5433244d4de64beae07f790bba140d3ba29337594 WatchSource:0}: Error finding container 809b9cd8a8cf9c692a9d5ea5433244d4de64beae07f790bba140d3ba29337594: Status 404 returned error can't find the container with id 809b9cd8a8cf9c692a9d5ea5433244d4de64beae07f790bba140d3ba29337594 Jul 13 22:23:35.317220 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:23:35.317203 2583 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 13 22:23:35.712240 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:23:35.712199 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-l8v2f" event={"ID":"bf682d14-b0e6-484f-bad2-ac5790938edf","Type":"ContainerStarted","Data":"8d178f59d645b46d5ff8f25c9175d8108b670f5fa50b4613e843363e8b58d141"} Jul 13 22:23:35.712403 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:23:35.712250 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-l8v2f" event={"ID":"bf682d14-b0e6-484f-bad2-ac5790938edf","Type":"ContainerStarted","Data":"809b9cd8a8cf9c692a9d5ea5433244d4de64beae07f790bba140d3ba29337594"} Jul 13 22:23:35.729447 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:23:35.729361 2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-l8v2f" podStartSLOduration=1.5687327770000001 podStartE2EDuration="1.729345258s" totalImagesPullingTime="160.612481ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-13 22:23:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 22:23:35.727674846 +0000 UTC m=+610.975590207" watchObservedRunningTime="2026-07-13 22:23:35.729345258 +0000 UTC m=+610.977260608" Jul 13 22:23:45.190078 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:23:45.190037 2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-l8v2f" Jul 13 22:23:45.190558 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:23:45.190129 2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-l8v2f" Jul 13 22:23:45.202613 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:23:45.202589 2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-l8v2f" Jul 13 22:23:45.763463 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:23:45.763437 2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-l8v2f" Jul 13 22:37:56.833052 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:37:56.833018 2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ait-d1d97912-openshift-default-7b85d48ddf-2nwtd"] Jul 13 22:37:56.835289 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:37:56.835260 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-d1d97912-openshift-default-7b85d48ddf-2nwtd" Jul 13 22:37:56.837439 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:37:56.837418 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ait-d1d97912-openshift-default-dockercfg-298b7\"" Jul 13 22:37:56.861237 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:37:56.859649 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-d1d97912-openshift-default-7b85d48ddf-2nwtd"] Jul 13 22:37:56.969069 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:37:56.969031 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7zbnq\" (UniqueName: \"kubernetes.io/projected/db37f4c7-fff5-491d-b3c4-b8f33ab5b2da-kube-api-access-7zbnq\") pod \"e2e-ait-d1d97912-openshift-default-7b85d48ddf-2nwtd\" (UID: \"db37f4c7-fff5-491d-b3c4-b8f33ab5b2da\") " pod="openshift-ingress/e2e-ait-d1d97912-openshift-default-7b85d48ddf-2nwtd" Jul 13 22:37:56.969069 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:37:56.969076 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/db37f4c7-fff5-491d-b3c4-b8f33ab5b2da-istio-podinfo\") pod \"e2e-ait-d1d97912-openshift-default-7b85d48ddf-2nwtd\" (UID: \"db37f4c7-fff5-491d-b3c4-b8f33ab5b2da\") " pod="openshift-ingress/e2e-ait-d1d97912-openshift-default-7b85d48ddf-2nwtd" Jul 13 22:37:56.969316 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:37:56.969104 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/db37f4c7-fff5-491d-b3c4-b8f33ab5b2da-workload-certs\") pod \"e2e-ait-d1d97912-openshift-default-7b85d48ddf-2nwtd\" (UID: \"db37f4c7-fff5-491d-b3c4-b8f33ab5b2da\") " pod="openshift-ingress/e2e-ait-d1d97912-openshift-default-7b85d48ddf-2nwtd" Jul 13 22:37:56.969316 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:37:56.969183 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/db37f4c7-fff5-491d-b3c4-b8f33ab5b2da-credential-socket\") pod \"e2e-ait-d1d97912-openshift-default-7b85d48ddf-2nwtd\" (UID: \"db37f4c7-fff5-491d-b3c4-b8f33ab5b2da\") " pod="openshift-ingress/e2e-ait-d1d97912-openshift-default-7b85d48ddf-2nwtd" Jul 13 22:37:56.969316 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:37:56.969252 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/db37f4c7-fff5-491d-b3c4-b8f33ab5b2da-istiod-ca-cert\") pod \"e2e-ait-d1d97912-openshift-default-7b85d48ddf-2nwtd\" (UID: \"db37f4c7-fff5-491d-b3c4-b8f33ab5b2da\") " pod="openshift-ingress/e2e-ait-d1d97912-openshift-default-7b85d48ddf-2nwtd" Jul 13 22:37:56.969316 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:37:56.969306 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/db37f4c7-fff5-491d-b3c4-b8f33ab5b2da-istio-token\") pod \"e2e-ait-d1d97912-openshift-default-7b85d48ddf-2nwtd\" (UID: \"db37f4c7-fff5-491d-b3c4-b8f33ab5b2da\") " pod="openshift-ingress/e2e-ait-d1d97912-openshift-default-7b85d48ddf-2nwtd" Jul 13 22:37:56.969442 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:37:56.969328 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/db37f4c7-fff5-491d-b3c4-b8f33ab5b2da-workload-socket\") pod \"e2e-ait-d1d97912-openshift-default-7b85d48ddf-2nwtd\" (UID: \"db37f4c7-fff5-491d-b3c4-b8f33ab5b2da\") " pod="openshift-ingress/e2e-ait-d1d97912-openshift-default-7b85d48ddf-2nwtd" Jul 13 22:37:56.969442 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:37:56.969366 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/db37f4c7-fff5-491d-b3c4-b8f33ab5b2da-istio-data\") pod \"e2e-ait-d1d97912-openshift-default-7b85d48ddf-2nwtd\" (UID: \"db37f4c7-fff5-491d-b3c4-b8f33ab5b2da\") " pod="openshift-ingress/e2e-ait-d1d97912-openshift-default-7b85d48ddf-2nwtd" Jul 13 22:37:56.969442 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:37:56.969388 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/db37f4c7-fff5-491d-b3c4-b8f33ab5b2da-istio-envoy\") pod \"e2e-ait-d1d97912-openshift-default-7b85d48ddf-2nwtd\" (UID: \"db37f4c7-fff5-491d-b3c4-b8f33ab5b2da\") " pod="openshift-ingress/e2e-ait-d1d97912-openshift-default-7b85d48ddf-2nwtd" Jul 13 22:37:57.070356 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:37:57.070313 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/db37f4c7-fff5-491d-b3c4-b8f33ab5b2da-workload-certs\") pod \"e2e-ait-d1d97912-openshift-default-7b85d48ddf-2nwtd\" (UID: \"db37f4c7-fff5-491d-b3c4-b8f33ab5b2da\") " pod="openshift-ingress/e2e-ait-d1d97912-openshift-default-7b85d48ddf-2nwtd" Jul 13 22:37:57.070356 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:37:57.070356 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/db37f4c7-fff5-491d-b3c4-b8f33ab5b2da-credential-socket\") pod \"e2e-ait-d1d97912-openshift-default-7b85d48ddf-2nwtd\" (UID: \"db37f4c7-fff5-491d-b3c4-b8f33ab5b2da\") " pod="openshift-ingress/e2e-ait-d1d97912-openshift-default-7b85d48ddf-2nwtd" Jul 13 22:37:57.070643 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:37:57.070376 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/db37f4c7-fff5-491d-b3c4-b8f33ab5b2da-istiod-ca-cert\") pod \"e2e-ait-d1d97912-openshift-default-7b85d48ddf-2nwtd\" (UID: \"db37f4c7-fff5-491d-b3c4-b8f33ab5b2da\") " pod="openshift-ingress/e2e-ait-d1d97912-openshift-default-7b85d48ddf-2nwtd" Jul 13 22:37:57.070643 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:37:57.070406 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/db37f4c7-fff5-491d-b3c4-b8f33ab5b2da-istio-token\") pod \"e2e-ait-d1d97912-openshift-default-7b85d48ddf-2nwtd\" (UID: \"db37f4c7-fff5-491d-b3c4-b8f33ab5b2da\") " pod="openshift-ingress/e2e-ait-d1d97912-openshift-default-7b85d48ddf-2nwtd" Jul 13 22:37:57.070643 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:37:57.070431 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/db37f4c7-fff5-491d-b3c4-b8f33ab5b2da-workload-socket\") pod \"e2e-ait-d1d97912-openshift-default-7b85d48ddf-2nwtd\" (UID: \"db37f4c7-fff5-491d-b3c4-b8f33ab5b2da\") " pod="openshift-ingress/e2e-ait-d1d97912-openshift-default-7b85d48ddf-2nwtd" Jul 13 22:37:57.070643 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:37:57.070456 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/db37f4c7-fff5-491d-b3c4-b8f33ab5b2da-istio-data\") pod \"e2e-ait-d1d97912-openshift-default-7b85d48ddf-2nwtd\" (UID: \"db37f4c7-fff5-491d-b3c4-b8f33ab5b2da\") " pod="openshift-ingress/e2e-ait-d1d97912-openshift-default-7b85d48ddf-2nwtd" Jul 13 22:37:57.070643 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:37:57.070473 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/db37f4c7-fff5-491d-b3c4-b8f33ab5b2da-istio-envoy\") pod \"e2e-ait-d1d97912-openshift-default-7b85d48ddf-2nwtd\" (UID: \"db37f4c7-fff5-491d-b3c4-b8f33ab5b2da\") " pod="openshift-ingress/e2e-ait-d1d97912-openshift-default-7b85d48ddf-2nwtd" Jul 13 22:37:57.070643 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:37:57.070531 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7zbnq\" (UniqueName: \"kubernetes.io/projected/db37f4c7-fff5-491d-b3c4-b8f33ab5b2da-kube-api-access-7zbnq\") pod \"e2e-ait-d1d97912-openshift-default-7b85d48ddf-2nwtd\" (UID: \"db37f4c7-fff5-491d-b3c4-b8f33ab5b2da\") " pod="openshift-ingress/e2e-ait-d1d97912-openshift-default-7b85d48ddf-2nwtd" Jul 13 22:37:57.070643 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:37:57.070562 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/db37f4c7-fff5-491d-b3c4-b8f33ab5b2da-istio-podinfo\") pod \"e2e-ait-d1d97912-openshift-default-7b85d48ddf-2nwtd\" (UID: \"db37f4c7-fff5-491d-b3c4-b8f33ab5b2da\") " pod="openshift-ingress/e2e-ait-d1d97912-openshift-default-7b85d48ddf-2nwtd" Jul 13 22:37:57.070969 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:37:57.070837 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/db37f4c7-fff5-491d-b3c4-b8f33ab5b2da-credential-socket\") pod \"e2e-ait-d1d97912-openshift-default-7b85d48ddf-2nwtd\" (UID: \"db37f4c7-fff5-491d-b3c4-b8f33ab5b2da\") " pod="openshift-ingress/e2e-ait-d1d97912-openshift-default-7b85d48ddf-2nwtd" Jul 13 22:37:57.070969 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:37:57.070888 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/db37f4c7-fff5-491d-b3c4-b8f33ab5b2da-istio-data\") pod \"e2e-ait-d1d97912-openshift-default-7b85d48ddf-2nwtd\" (UID: \"db37f4c7-fff5-491d-b3c4-b8f33ab5b2da\") " pod="openshift-ingress/e2e-ait-d1d97912-openshift-default-7b85d48ddf-2nwtd" Jul 13 22:37:57.070969 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:37:57.070933 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/db37f4c7-fff5-491d-b3c4-b8f33ab5b2da-workload-socket\") pod \"e2e-ait-d1d97912-openshift-default-7b85d48ddf-2nwtd\" (UID: \"db37f4c7-fff5-491d-b3c4-b8f33ab5b2da\") " pod="openshift-ingress/e2e-ait-d1d97912-openshift-default-7b85d48ddf-2nwtd" Jul 13 22:37:57.071114 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:37:57.071088 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/db37f4c7-fff5-491d-b3c4-b8f33ab5b2da-workload-certs\") pod \"e2e-ait-d1d97912-openshift-default-7b85d48ddf-2nwtd\" (UID: \"db37f4c7-fff5-491d-b3c4-b8f33ab5b2da\") " pod="openshift-ingress/e2e-ait-d1d97912-openshift-default-7b85d48ddf-2nwtd" Jul 13 22:37:57.071196 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:37:57.071177 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/db37f4c7-fff5-491d-b3c4-b8f33ab5b2da-istiod-ca-cert\") pod \"e2e-ait-d1d97912-openshift-default-7b85d48ddf-2nwtd\" (UID: \"db37f4c7-fff5-491d-b3c4-b8f33ab5b2da\") " pod="openshift-ingress/e2e-ait-d1d97912-openshift-default-7b85d48ddf-2nwtd" Jul 13 22:37:57.072872 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:37:57.072854 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/db37f4c7-fff5-491d-b3c4-b8f33ab5b2da-istio-envoy\") pod \"e2e-ait-d1d97912-openshift-default-7b85d48ddf-2nwtd\" (UID: \"db37f4c7-fff5-491d-b3c4-b8f33ab5b2da\") " pod="openshift-ingress/e2e-ait-d1d97912-openshift-default-7b85d48ddf-2nwtd" Jul 13 22:37:57.073078 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:37:57.073061 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/db37f4c7-fff5-491d-b3c4-b8f33ab5b2da-istio-podinfo\") pod \"e2e-ait-d1d97912-openshift-default-7b85d48ddf-2nwtd\" (UID: \"db37f4c7-fff5-491d-b3c4-b8f33ab5b2da\") " pod="openshift-ingress/e2e-ait-d1d97912-openshift-default-7b85d48ddf-2nwtd" Jul 13 22:37:57.079466 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:37:57.079443 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7zbnq\" (UniqueName: \"kubernetes.io/projected/db37f4c7-fff5-491d-b3c4-b8f33ab5b2da-kube-api-access-7zbnq\") pod \"e2e-ait-d1d97912-openshift-default-7b85d48ddf-2nwtd\" (UID: \"db37f4c7-fff5-491d-b3c4-b8f33ab5b2da\") " pod="openshift-ingress/e2e-ait-d1d97912-openshift-default-7b85d48ddf-2nwtd" Jul 13 22:37:57.081138 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:37:57.081118 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/db37f4c7-fff5-491d-b3c4-b8f33ab5b2da-istio-token\") pod \"e2e-ait-d1d97912-openshift-default-7b85d48ddf-2nwtd\" (UID: \"db37f4c7-fff5-491d-b3c4-b8f33ab5b2da\") " pod="openshift-ingress/e2e-ait-d1d97912-openshift-default-7b85d48ddf-2nwtd" Jul 13 22:37:57.146624 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:37:57.146515 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-d1d97912-openshift-default-7b85d48ddf-2nwtd" Jul 13 22:37:57.279853 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:37:57.279819 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-d1d97912-openshift-default-7b85d48ddf-2nwtd"] Jul 13 22:37:57.283148 ip-10-0-133-179 kubenswrapper[2583]: W0713 22:37:57.283126 2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddb37f4c7_fff5_491d_b3c4_b8f33ab5b2da.slice/crio-e74ea1ae7e6ce75dfa54fc1068bec90b1bcbedb6793a9748e8fe2e17e63c4984 WatchSource:0}: Error finding container e74ea1ae7e6ce75dfa54fc1068bec90b1bcbedb6793a9748e8fe2e17e63c4984: Status 404 returned error can't find the container with id e74ea1ae7e6ce75dfa54fc1068bec90b1bcbedb6793a9748e8fe2e17e63c4984 Jul 13 22:37:57.285201 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:37:57.285163 2583 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 13 22:37:57.285255 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:37:57.285238 2583 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 13 22:37:57.285288 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:37:57.285270 2583 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 13 22:37:58.043907 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:37:58.043876 2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-d1d97912-56dd4466bb-2t8np"] Jul 13 22:37:58.046342 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:37:58.046320 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-d1d97912-56dd4466bb-2t8np" Jul 13 22:37:58.048445 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:37:58.048420 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ait-d1d97912-serving-cert\"" Jul 13 22:37:58.058208 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:37:58.058184 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-d1d97912-56dd4466bb-2t8np"] Jul 13 22:37:58.139133 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:37:58.139097 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-d1d97912-openshift-default-7b85d48ddf-2nwtd" event={"ID":"db37f4c7-fff5-491d-b3c4-b8f33ab5b2da","Type":"ContainerStarted","Data":"e2afe28fcf6ec5aad0e0624e5acce08441f473fafd4066c1e18261f47517e562"} Jul 13 22:37:58.139305 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:37:58.139153 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-d1d97912-openshift-default-7b85d48ddf-2nwtd" event={"ID":"db37f4c7-fff5-491d-b3c4-b8f33ab5b2da","Type":"ContainerStarted","Data":"e74ea1ae7e6ce75dfa54fc1068bec90b1bcbedb6793a9748e8fe2e17e63c4984"} Jul 13 22:37:58.146909 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:37:58.146882 2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-d1d97912-openshift-default-7b85d48ddf-2nwtd" Jul 13 22:37:58.151939 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:37:58.151915 2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-d1d97912-openshift-default-7b85d48ddf-2nwtd" Jul 13 22:37:58.159418 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:37:58.159376 2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-ait-d1d97912-openshift-default-7b85d48ddf-2nwtd" podStartSLOduration=2.159361517 podStartE2EDuration="2.159361517s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-13 22:37:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 22:37:58.158064682 +0000 UTC m=+1473.405980033" watchObservedRunningTime="2026-07-13 22:37:58.159361517 +0000 UTC m=+1473.407276866" Jul 13 22:37:58.181738 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:37:58.181691 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1e2d553c-a08f-48e5-b792-a77ce98533bd-maas-api-tls\") pod \"maas-api-e2e-ait-d1d97912-56dd4466bb-2t8np\" (UID: \"1e2d553c-a08f-48e5-b792-a77ce98533bd\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-d1d97912-56dd4466bb-2t8np" Jul 13 22:37:58.181933 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:37:58.181779 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1e2d553c-a08f-48e5-b792-a77ce98533bd-openshift-service-ca\") pod \"maas-api-e2e-ait-d1d97912-56dd4466bb-2t8np\" (UID: \"1e2d553c-a08f-48e5-b792-a77ce98533bd\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-d1d97912-56dd4466bb-2t8np" Jul 13 22:37:58.181933 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:37:58.181841 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zvkkf\" (UniqueName: \"kubernetes.io/projected/1e2d553c-a08f-48e5-b792-a77ce98533bd-kube-api-access-zvkkf\") pod \"maas-api-e2e-ait-d1d97912-56dd4466bb-2t8np\" (UID: \"1e2d553c-a08f-48e5-b792-a77ce98533bd\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-d1d97912-56dd4466bb-2t8np" Jul 13 22:37:58.282681 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:37:58.282625 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zvkkf\" (UniqueName: \"kubernetes.io/projected/1e2d553c-a08f-48e5-b792-a77ce98533bd-kube-api-access-zvkkf\") pod \"maas-api-e2e-ait-d1d97912-56dd4466bb-2t8np\" (UID: \"1e2d553c-a08f-48e5-b792-a77ce98533bd\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-d1d97912-56dd4466bb-2t8np" Jul 13 22:37:58.282879 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:37:58.282742 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1e2d553c-a08f-48e5-b792-a77ce98533bd-maas-api-tls\") pod \"maas-api-e2e-ait-d1d97912-56dd4466bb-2t8np\" (UID: \"1e2d553c-a08f-48e5-b792-a77ce98533bd\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-d1d97912-56dd4466bb-2t8np" Jul 13 22:37:58.282879 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:37:58.282833 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1e2d553c-a08f-48e5-b792-a77ce98533bd-openshift-service-ca\") pod \"maas-api-e2e-ait-d1d97912-56dd4466bb-2t8np\" (UID: \"1e2d553c-a08f-48e5-b792-a77ce98533bd\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-d1d97912-56dd4466bb-2t8np" Jul 13 22:37:58.283536 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:37:58.283511 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1e2d553c-a08f-48e5-b792-a77ce98533bd-openshift-service-ca\") pod \"maas-api-e2e-ait-d1d97912-56dd4466bb-2t8np\" (UID: \"1e2d553c-a08f-48e5-b792-a77ce98533bd\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-d1d97912-56dd4466bb-2t8np" Jul 13 22:37:58.285496 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:37:58.285466 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1e2d553c-a08f-48e5-b792-a77ce98533bd-maas-api-tls\") pod \"maas-api-e2e-ait-d1d97912-56dd4466bb-2t8np\" (UID: \"1e2d553c-a08f-48e5-b792-a77ce98533bd\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-d1d97912-56dd4466bb-2t8np" Jul 13 22:37:58.291457 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:37:58.291420 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zvkkf\" (UniqueName: \"kubernetes.io/projected/1e2d553c-a08f-48e5-b792-a77ce98533bd-kube-api-access-zvkkf\") pod \"maas-api-e2e-ait-d1d97912-56dd4466bb-2t8np\" (UID: \"1e2d553c-a08f-48e5-b792-a77ce98533bd\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-d1d97912-56dd4466bb-2t8np" Jul 13 22:37:58.358368 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:37:58.358270 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-d1d97912-56dd4466bb-2t8np" Jul 13 22:37:58.497155 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:37:58.497124 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-d1d97912-56dd4466bb-2t8np"] Jul 13 22:37:58.500785 ip-10-0-133-179 kubenswrapper[2583]: W0713 22:37:58.500758 2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1e2d553c_a08f_48e5_b792_a77ce98533bd.slice/crio-66bb105b16797da24feb4a6a15441a274bc082479c1e28d9a68bb37de1571216 WatchSource:0}: Error finding container 66bb105b16797da24feb4a6a15441a274bc082479c1e28d9a68bb37de1571216: Status 404 returned error can't find the container with id 66bb105b16797da24feb4a6a15441a274bc082479c1e28d9a68bb37de1571216 Jul 13 22:37:58.502214 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:37:58.502196 2583 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 13 22:37:59.145233 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:37:59.145190 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-d1d97912-56dd4466bb-2t8np" event={"ID":"1e2d553c-a08f-48e5-b792-a77ce98533bd","Type":"ContainerStarted","Data":"66bb105b16797da24feb4a6a15441a274bc082479c1e28d9a68bb37de1571216"} Jul 13 22:37:59.145634 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:37:59.145481 2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-d1d97912-openshift-default-7b85d48ddf-2nwtd" Jul 13 22:37:59.146501 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:37:59.146483 2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-d1d97912-openshift-default-7b85d48ddf-2nwtd" Jul 13 22:38:00.150948 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:38:00.150908 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-d1d97912-56dd4466bb-2t8np" event={"ID":"1e2d553c-a08f-48e5-b792-a77ce98533bd","Type":"ContainerStarted","Data":"0c0f7a0118cb2edd456ae19e6bec26d8c4786c8dfa8e196430ab2382704fc723"} Jul 13 22:38:00.169856 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:38:00.169805 2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ait-d1d97912-56dd4466bb-2t8np" podStartSLOduration=1.578158483 podStartE2EDuration="2.169786538s" totalImagesPullingTime="591.628055ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-13 22:37:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 22:38:00.167807664 +0000 UTC m=+1475.415723027" watchObservedRunningTime="2026-07-13 22:38:00.169786538 +0000 UTC m=+1475.417701925" Jul 13 22:38:01.154892 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:38:01.154856 2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-d1d97912-56dd4466bb-2t8np" Jul 13 22:38:03.280543 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:38:03.280499 2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-d1d97912-openshift-default-7b85d48ddf-2nwtd"] Jul 13 22:38:03.280986 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:38:03.280720 2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-ait-d1d97912-openshift-default-7b85d48ddf-2nwtd" podUID="db37f4c7-fff5-491d-b3c4-b8f33ab5b2da" containerName="istio-proxy" containerID="cri-o://e2afe28fcf6ec5aad0e0624e5acce08441f473fafd4066c1e18261f47517e562" gracePeriod=30 Jul 13 22:38:03.464662 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:38:03.464628 2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-d1d97912-56dd4466bb-2t8np"] Jul 13 22:38:03.464872 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:38:03.464847 2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ait-d1d97912-56dd4466bb-2t8np" podUID="1e2d553c-a08f-48e5-b792-a77ce98533bd" containerName="maas-api" containerID="cri-o://0c0f7a0118cb2edd456ae19e6bec26d8c4786c8dfa8e196430ab2382704fc723" gracePeriod=30 Jul 13 22:38:03.720253 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:38:03.720228 2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-d1d97912-56dd4466bb-2t8np" Jul 13 22:38:03.845642 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:38:03.840852 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1e2d553c-a08f-48e5-b792-a77ce98533bd-openshift-service-ca\") pod \"1e2d553c-a08f-48e5-b792-a77ce98533bd\" (UID: \"1e2d553c-a08f-48e5-b792-a77ce98533bd\") " Jul 13 22:38:03.845642 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:38:03.840909 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1e2d553c-a08f-48e5-b792-a77ce98533bd-maas-api-tls\") pod \"1e2d553c-a08f-48e5-b792-a77ce98533bd\" (UID: \"1e2d553c-a08f-48e5-b792-a77ce98533bd\") " Jul 13 22:38:03.845642 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:38:03.841032 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-zvkkf\" (UniqueName: \"kubernetes.io/projected/1e2d553c-a08f-48e5-b792-a77ce98533bd-kube-api-access-zvkkf\") pod \"1e2d553c-a08f-48e5-b792-a77ce98533bd\" (UID: \"1e2d553c-a08f-48e5-b792-a77ce98533bd\") " Jul 13 22:38:03.845642 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:38:03.841912 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1e2d553c-a08f-48e5-b792-a77ce98533bd-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "1e2d553c-a08f-48e5-b792-a77ce98533bd" (UID: "1e2d553c-a08f-48e5-b792-a77ce98533bd"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 13 22:38:03.846593 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:38:03.846472 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1e2d553c-a08f-48e5-b792-a77ce98533bd-kube-api-access-zvkkf" (OuterVolumeSpecName: "kube-api-access-zvkkf") pod "1e2d553c-a08f-48e5-b792-a77ce98533bd" (UID: "1e2d553c-a08f-48e5-b792-a77ce98533bd"). InnerVolumeSpecName "kube-api-access-zvkkf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 13 22:38:03.850592 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:38:03.846917 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1e2d553c-a08f-48e5-b792-a77ce98533bd-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "1e2d553c-a08f-48e5-b792-a77ce98533bd" (UID: "1e2d553c-a08f-48e5-b792-a77ce98533bd"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 13 22:38:03.942735 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:38:03.942646 2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zvkkf\" (UniqueName: \"kubernetes.io/projected/1e2d553c-a08f-48e5-b792-a77ce98533bd-kube-api-access-zvkkf\") on node \"ip-10-0-133-179.ec2.internal\" DevicePath \"\"" Jul 13 22:38:03.942735 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:38:03.942685 2583 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1e2d553c-a08f-48e5-b792-a77ce98533bd-openshift-service-ca\") on node \"ip-10-0-133-179.ec2.internal\" DevicePath \"\"" Jul 13 22:38:03.942735 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:38:03.942705 2583 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1e2d553c-a08f-48e5-b792-a77ce98533bd-maas-api-tls\") on node \"ip-10-0-133-179.ec2.internal\" DevicePath \"\"" Jul 13 22:38:04.169303 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:38:04.169209 2583 generic.go:356] "Generic (PLEG): container finished" podID="1e2d553c-a08f-48e5-b792-a77ce98533bd" containerID="0c0f7a0118cb2edd456ae19e6bec26d8c4786c8dfa8e196430ab2382704fc723" exitCode=0 Jul 13 22:38:04.169303 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:38:04.169257 2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-d1d97912-56dd4466bb-2t8np" Jul 13 22:38:04.169303 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:38:04.169285 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-d1d97912-56dd4466bb-2t8np" event={"ID":"1e2d553c-a08f-48e5-b792-a77ce98533bd","Type":"ContainerDied","Data":"0c0f7a0118cb2edd456ae19e6bec26d8c4786c8dfa8e196430ab2382704fc723"} Jul 13 22:38:04.169520 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:38:04.169338 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-d1d97912-56dd4466bb-2t8np" event={"ID":"1e2d553c-a08f-48e5-b792-a77ce98533bd","Type":"ContainerDied","Data":"66bb105b16797da24feb4a6a15441a274bc082479c1e28d9a68bb37de1571216"} Jul 13 22:38:04.169520 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:38:04.169369 2583 scope.go:117] "RemoveContainer" containerID="0c0f7a0118cb2edd456ae19e6bec26d8c4786c8dfa8e196430ab2382704fc723" Jul 13 22:38:04.177975 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:38:04.177954 2583 scope.go:117] "RemoveContainer" containerID="0c0f7a0118cb2edd456ae19e6bec26d8c4786c8dfa8e196430ab2382704fc723" Jul 13 22:38:04.178286 ip-10-0-133-179 kubenswrapper[2583]: E0713 22:38:04.178267 2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0c0f7a0118cb2edd456ae19e6bec26d8c4786c8dfa8e196430ab2382704fc723\": container with ID starting with 0c0f7a0118cb2edd456ae19e6bec26d8c4786c8dfa8e196430ab2382704fc723 not found: ID does not exist" containerID="0c0f7a0118cb2edd456ae19e6bec26d8c4786c8dfa8e196430ab2382704fc723" Jul 13 22:38:04.178335 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:38:04.178298 2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0c0f7a0118cb2edd456ae19e6bec26d8c4786c8dfa8e196430ab2382704fc723"} err="failed to get container status \"0c0f7a0118cb2edd456ae19e6bec26d8c4786c8dfa8e196430ab2382704fc723\": rpc error: code = NotFound desc = could not find container \"0c0f7a0118cb2edd456ae19e6bec26d8c4786c8dfa8e196430ab2382704fc723\": container with ID starting with 0c0f7a0118cb2edd456ae19e6bec26d8c4786c8dfa8e196430ab2382704fc723 not found: ID does not exist" Jul 13 22:38:04.189129 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:38:04.189098 2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-d1d97912-56dd4466bb-2t8np"] Jul 13 22:38:04.193282 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:38:04.193258 2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-d1d97912-56dd4466bb-2t8np"] Jul 13 22:38:05.389043 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:38:05.389011 2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1e2d553c-a08f-48e5-b792-a77ce98533bd" path="/var/lib/kubelet/pods/1e2d553c-a08f-48e5-b792-a77ce98533bd/volumes" Jul 13 22:38:09.144023 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:38:09.143992 2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-d1d97912-openshift-default-7b85d48ddf-2nwtd" Jul 13 22:38:09.194372 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:38:09.194284 2583 generic.go:356] "Generic (PLEG): container finished" podID="db37f4c7-fff5-491d-b3c4-b8f33ab5b2da" containerID="e2afe28fcf6ec5aad0e0624e5acce08441f473fafd4066c1e18261f47517e562" exitCode=0 Jul 13 22:38:09.194372 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:38:09.194345 2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-d1d97912-openshift-default-7b85d48ddf-2nwtd" Jul 13 22:38:09.194549 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:38:09.194360 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-d1d97912-openshift-default-7b85d48ddf-2nwtd" event={"ID":"db37f4c7-fff5-491d-b3c4-b8f33ab5b2da","Type":"ContainerDied","Data":"e2afe28fcf6ec5aad0e0624e5acce08441f473fafd4066c1e18261f47517e562"} Jul 13 22:38:09.194549 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:38:09.194425 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-d1d97912-openshift-default-7b85d48ddf-2nwtd" event={"ID":"db37f4c7-fff5-491d-b3c4-b8f33ab5b2da","Type":"ContainerDied","Data":"e74ea1ae7e6ce75dfa54fc1068bec90b1bcbedb6793a9748e8fe2e17e63c4984"} Jul 13 22:38:09.194549 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:38:09.194449 2583 scope.go:117] "RemoveContainer" containerID="e2afe28fcf6ec5aad0e0624e5acce08441f473fafd4066c1e18261f47517e562" Jul 13 22:38:09.202907 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:38:09.202886 2583 scope.go:117] "RemoveContainer" containerID="e2afe28fcf6ec5aad0e0624e5acce08441f473fafd4066c1e18261f47517e562" Jul 13 22:38:09.203239 ip-10-0-133-179 kubenswrapper[2583]: E0713 22:38:09.203220 2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e2afe28fcf6ec5aad0e0624e5acce08441f473fafd4066c1e18261f47517e562\": container with ID starting with e2afe28fcf6ec5aad0e0624e5acce08441f473fafd4066c1e18261f47517e562 not found: ID does not exist" containerID="e2afe28fcf6ec5aad0e0624e5acce08441f473fafd4066c1e18261f47517e562" Jul 13 22:38:09.203295 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:38:09.203247 2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e2afe28fcf6ec5aad0e0624e5acce08441f473fafd4066c1e18261f47517e562"} err="failed to get container status \"e2afe28fcf6ec5aad0e0624e5acce08441f473fafd4066c1e18261f47517e562\": rpc error: code = NotFound desc = could not find container \"e2afe28fcf6ec5aad0e0624e5acce08441f473fafd4066c1e18261f47517e562\": container with ID starting with e2afe28fcf6ec5aad0e0624e5acce08441f473fafd4066c1e18261f47517e562 not found: ID does not exist" Jul 13 22:38:09.295456 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:38:09.295415 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/db37f4c7-fff5-491d-b3c4-b8f33ab5b2da-istio-token\") pod \"db37f4c7-fff5-491d-b3c4-b8f33ab5b2da\" (UID: \"db37f4c7-fff5-491d-b3c4-b8f33ab5b2da\") " Jul 13 22:38:09.295632 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:38:09.295498 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/db37f4c7-fff5-491d-b3c4-b8f33ab5b2da-workload-certs\") pod \"db37f4c7-fff5-491d-b3c4-b8f33ab5b2da\" (UID: \"db37f4c7-fff5-491d-b3c4-b8f33ab5b2da\") " Jul 13 22:38:09.295632 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:38:09.295539 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/db37f4c7-fff5-491d-b3c4-b8f33ab5b2da-istiod-ca-cert\") pod \"db37f4c7-fff5-491d-b3c4-b8f33ab5b2da\" (UID: \"db37f4c7-fff5-491d-b3c4-b8f33ab5b2da\") " Jul 13 22:38:09.295632 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:38:09.295556 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7zbnq\" (UniqueName: \"kubernetes.io/projected/db37f4c7-fff5-491d-b3c4-b8f33ab5b2da-kube-api-access-7zbnq\") pod \"db37f4c7-fff5-491d-b3c4-b8f33ab5b2da\" (UID: \"db37f4c7-fff5-491d-b3c4-b8f33ab5b2da\") " Jul 13 22:38:09.295632 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:38:09.295621 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/db37f4c7-fff5-491d-b3c4-b8f33ab5b2da-istio-podinfo\") pod \"db37f4c7-fff5-491d-b3c4-b8f33ab5b2da\" (UID: \"db37f4c7-fff5-491d-b3c4-b8f33ab5b2da\") " Jul 13 22:38:09.295823 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:38:09.295645 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/db37f4c7-fff5-491d-b3c4-b8f33ab5b2da-credential-socket\") pod \"db37f4c7-fff5-491d-b3c4-b8f33ab5b2da\" (UID: \"db37f4c7-fff5-491d-b3c4-b8f33ab5b2da\") " Jul 13 22:38:09.295823 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:38:09.295682 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/db37f4c7-fff5-491d-b3c4-b8f33ab5b2da-istio-envoy\") pod \"db37f4c7-fff5-491d-b3c4-b8f33ab5b2da\" (UID: \"db37f4c7-fff5-491d-b3c4-b8f33ab5b2da\") " Jul 13 22:38:09.295823 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:38:09.295725 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/db37f4c7-fff5-491d-b3c4-b8f33ab5b2da-istio-data\") pod \"db37f4c7-fff5-491d-b3c4-b8f33ab5b2da\" (UID: \"db37f4c7-fff5-491d-b3c4-b8f33ab5b2da\") " Jul 13 22:38:09.295823 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:38:09.295750 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/db37f4c7-fff5-491d-b3c4-b8f33ab5b2da-workload-socket\") pod \"db37f4c7-fff5-491d-b3c4-b8f33ab5b2da\" (UID: \"db37f4c7-fff5-491d-b3c4-b8f33ab5b2da\") " Jul 13 22:38:09.296016 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:38:09.295869 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/db37f4c7-fff5-491d-b3c4-b8f33ab5b2da-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "db37f4c7-fff5-491d-b3c4-b8f33ab5b2da" (UID: "db37f4c7-fff5-491d-b3c4-b8f33ab5b2da"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 13 22:38:09.296016 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:38:09.296002 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/db37f4c7-fff5-491d-b3c4-b8f33ab5b2da-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "db37f4c7-fff5-491d-b3c4-b8f33ab5b2da" (UID: "db37f4c7-fff5-491d-b3c4-b8f33ab5b2da"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 13 22:38:09.296112 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:38:09.296012 2583 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/db37f4c7-fff5-491d-b3c4-b8f33ab5b2da-workload-certs\") on node \"ip-10-0-133-179.ec2.internal\" DevicePath \"\"" Jul 13 22:38:09.296219 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:38:09.296199 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/db37f4c7-fff5-491d-b3c4-b8f33ab5b2da-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "db37f4c7-fff5-491d-b3c4-b8f33ab5b2da" (UID: "db37f4c7-fff5-491d-b3c4-b8f33ab5b2da"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 13 22:38:09.296342 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:38:09.296282 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/db37f4c7-fff5-491d-b3c4-b8f33ab5b2da-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "db37f4c7-fff5-491d-b3c4-b8f33ab5b2da" (UID: "db37f4c7-fff5-491d-b3c4-b8f33ab5b2da"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 13 22:38:09.296781 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:38:09.296756 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/db37f4c7-fff5-491d-b3c4-b8f33ab5b2da-istio-data" (OuterVolumeSpecName: "istio-data") pod "db37f4c7-fff5-491d-b3c4-b8f33ab5b2da" (UID: "db37f4c7-fff5-491d-b3c4-b8f33ab5b2da"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 13 22:38:09.298042 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:38:09.298021 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/db37f4c7-fff5-491d-b3c4-b8f33ab5b2da-kube-api-access-7zbnq" (OuterVolumeSpecName: "kube-api-access-7zbnq") pod "db37f4c7-fff5-491d-b3c4-b8f33ab5b2da" (UID: "db37f4c7-fff5-491d-b3c4-b8f33ab5b2da"). InnerVolumeSpecName "kube-api-access-7zbnq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 13 22:38:09.298202 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:38:09.298044 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/db37f4c7-fff5-491d-b3c4-b8f33ab5b2da-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "db37f4c7-fff5-491d-b3c4-b8f33ab5b2da" (UID: "db37f4c7-fff5-491d-b3c4-b8f33ab5b2da"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 13 22:38:09.298202 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:38:09.298138 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/db37f4c7-fff5-491d-b3c4-b8f33ab5b2da-istio-token" (OuterVolumeSpecName: "istio-token") pod "db37f4c7-fff5-491d-b3c4-b8f33ab5b2da" (UID: "db37f4c7-fff5-491d-b3c4-b8f33ab5b2da"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 13 22:38:09.298202 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:38:09.298133 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/db37f4c7-fff5-491d-b3c4-b8f33ab5b2da-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "db37f4c7-fff5-491d-b3c4-b8f33ab5b2da" (UID: "db37f4c7-fff5-491d-b3c4-b8f33ab5b2da"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 13 22:38:09.396755 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:38:09.396722 2583 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/db37f4c7-fff5-491d-b3c4-b8f33ab5b2da-istiod-ca-cert\") on node \"ip-10-0-133-179.ec2.internal\" DevicePath \"\"" Jul 13 22:38:09.396755 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:38:09.396751 2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7zbnq\" (UniqueName: \"kubernetes.io/projected/db37f4c7-fff5-491d-b3c4-b8f33ab5b2da-kube-api-access-7zbnq\") on node \"ip-10-0-133-179.ec2.internal\" DevicePath \"\"" Jul 13 22:38:09.396986 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:38:09.396762 2583 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/db37f4c7-fff5-491d-b3c4-b8f33ab5b2da-istio-podinfo\") on node \"ip-10-0-133-179.ec2.internal\" DevicePath \"\"" Jul 13 22:38:09.396986 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:38:09.396771 2583 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/db37f4c7-fff5-491d-b3c4-b8f33ab5b2da-credential-socket\") on node \"ip-10-0-133-179.ec2.internal\" DevicePath \"\"" Jul 13 22:38:09.396986 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:38:09.396780 2583 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/db37f4c7-fff5-491d-b3c4-b8f33ab5b2da-istio-envoy\") on node \"ip-10-0-133-179.ec2.internal\" DevicePath \"\"" Jul 13 22:38:09.396986 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:38:09.396789 2583 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/db37f4c7-fff5-491d-b3c4-b8f33ab5b2da-istio-data\") on node \"ip-10-0-133-179.ec2.internal\" DevicePath \"\"" Jul 13 22:38:09.396986 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:38:09.396797 2583 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/db37f4c7-fff5-491d-b3c4-b8f33ab5b2da-workload-socket\") on node \"ip-10-0-133-179.ec2.internal\" DevicePath \"\"" Jul 13 22:38:09.396986 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:38:09.396805 2583 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/db37f4c7-fff5-491d-b3c4-b8f33ab5b2da-istio-token\") on node \"ip-10-0-133-179.ec2.internal\" DevicePath \"\"" Jul 13 22:38:09.521757 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:38:09.521723 2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-d1d97912-openshift-default-7b85d48ddf-2nwtd"] Jul 13 22:38:09.529220 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:38:09.529190 2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ait-d1d97912-openshift-default-7b85d48ddf-2nwtd"] Jul 13 22:38:09.897939 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:38:09.897862 2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ait-429de0b9-openshift-default-679c445bc7-2d549"] Jul 13 22:38:09.898266 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:38:09.898251 2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="db37f4c7-fff5-491d-b3c4-b8f33ab5b2da" containerName="istio-proxy" Jul 13 22:38:09.898318 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:38:09.898270 2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="db37f4c7-fff5-491d-b3c4-b8f33ab5b2da" containerName="istio-proxy" Jul 13 22:38:09.898318 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:38:09.898284 2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1e2d553c-a08f-48e5-b792-a77ce98533bd" containerName="maas-api" Jul 13 22:38:09.898318 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:38:09.898290 2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="1e2d553c-a08f-48e5-b792-a77ce98533bd" containerName="maas-api" Jul 13 22:38:09.898415 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:38:09.898355 2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="1e2d553c-a08f-48e5-b792-a77ce98533bd" containerName="maas-api" Jul 13 22:38:09.898415 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:38:09.898364 2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="db37f4c7-fff5-491d-b3c4-b8f33ab5b2da" containerName="istio-proxy" Jul 13 22:38:09.907485 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:38:09.907460 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-429de0b9-openshift-default-679c445bc7-2d549" Jul 13 22:38:09.915337 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:38:09.911253 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ait-429de0b9-openshift-default-dockercfg-v85w2\"" Jul 13 22:38:09.917195 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:38:09.917117 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-429de0b9-openshift-default-679c445bc7-2d549"] Jul 13 22:38:10.002316 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:38:10.002275 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/df056db4-2e39-44b9-8fa6-d50e1d7abf65-workload-socket\") pod \"e2e-ait-429de0b9-openshift-default-679c445bc7-2d549\" (UID: \"df056db4-2e39-44b9-8fa6-d50e1d7abf65\") " pod="openshift-ingress/e2e-ait-429de0b9-openshift-default-679c445bc7-2d549" Jul 13 22:38:10.002316 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:38:10.002319 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/df056db4-2e39-44b9-8fa6-d50e1d7abf65-istio-envoy\") pod \"e2e-ait-429de0b9-openshift-default-679c445bc7-2d549\" (UID: \"df056db4-2e39-44b9-8fa6-d50e1d7abf65\") " pod="openshift-ingress/e2e-ait-429de0b9-openshift-default-679c445bc7-2d549" Jul 13 22:38:10.002554 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:38:10.002342 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/df056db4-2e39-44b9-8fa6-d50e1d7abf65-istio-podinfo\") pod \"e2e-ait-429de0b9-openshift-default-679c445bc7-2d549\" (UID: \"df056db4-2e39-44b9-8fa6-d50e1d7abf65\") " pod="openshift-ingress/e2e-ait-429de0b9-openshift-default-679c445bc7-2d549" Jul 13 22:38:10.002554 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:38:10.002426 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/df056db4-2e39-44b9-8fa6-d50e1d7abf65-istiod-ca-cert\") pod \"e2e-ait-429de0b9-openshift-default-679c445bc7-2d549\" (UID: \"df056db4-2e39-44b9-8fa6-d50e1d7abf65\") " pod="openshift-ingress/e2e-ait-429de0b9-openshift-default-679c445bc7-2d549" Jul 13 22:38:10.002554 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:38:10.002501 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/df056db4-2e39-44b9-8fa6-d50e1d7abf65-credential-socket\") pod \"e2e-ait-429de0b9-openshift-default-679c445bc7-2d549\" (UID: \"df056db4-2e39-44b9-8fa6-d50e1d7abf65\") " pod="openshift-ingress/e2e-ait-429de0b9-openshift-default-679c445bc7-2d549" Jul 13 22:38:10.002554 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:38:10.002522 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/df056db4-2e39-44b9-8fa6-d50e1d7abf65-istio-token\") pod \"e2e-ait-429de0b9-openshift-default-679c445bc7-2d549\" (UID: \"df056db4-2e39-44b9-8fa6-d50e1d7abf65\") " pod="openshift-ingress/e2e-ait-429de0b9-openshift-default-679c445bc7-2d549" Jul 13 22:38:10.002738 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:38:10.002554 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j8rll\" (UniqueName: \"kubernetes.io/projected/df056db4-2e39-44b9-8fa6-d50e1d7abf65-kube-api-access-j8rll\") pod \"e2e-ait-429de0b9-openshift-default-679c445bc7-2d549\" (UID: \"df056db4-2e39-44b9-8fa6-d50e1d7abf65\") " pod="openshift-ingress/e2e-ait-429de0b9-openshift-default-679c445bc7-2d549" Jul 13 22:38:10.002738 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:38:10.002608 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/df056db4-2e39-44b9-8fa6-d50e1d7abf65-workload-certs\") pod \"e2e-ait-429de0b9-openshift-default-679c445bc7-2d549\" (UID: \"df056db4-2e39-44b9-8fa6-d50e1d7abf65\") " pod="openshift-ingress/e2e-ait-429de0b9-openshift-default-679c445bc7-2d549" Jul 13 22:38:10.002738 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:38:10.002639 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/df056db4-2e39-44b9-8fa6-d50e1d7abf65-istio-data\") pod \"e2e-ait-429de0b9-openshift-default-679c445bc7-2d549\" (UID: \"df056db4-2e39-44b9-8fa6-d50e1d7abf65\") " pod="openshift-ingress/e2e-ait-429de0b9-openshift-default-679c445bc7-2d549" Jul 13 22:38:10.103560 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:38:10.103501 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j8rll\" (UniqueName: \"kubernetes.io/projected/df056db4-2e39-44b9-8fa6-d50e1d7abf65-kube-api-access-j8rll\") pod \"e2e-ait-429de0b9-openshift-default-679c445bc7-2d549\" (UID: \"df056db4-2e39-44b9-8fa6-d50e1d7abf65\") " pod="openshift-ingress/e2e-ait-429de0b9-openshift-default-679c445bc7-2d549" Jul 13 22:38:10.103560 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:38:10.103600 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/df056db4-2e39-44b9-8fa6-d50e1d7abf65-workload-certs\") pod \"e2e-ait-429de0b9-openshift-default-679c445bc7-2d549\" (UID: \"df056db4-2e39-44b9-8fa6-d50e1d7abf65\") " pod="openshift-ingress/e2e-ait-429de0b9-openshift-default-679c445bc7-2d549" Jul 13 22:38:10.103873 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:38:10.103643 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/df056db4-2e39-44b9-8fa6-d50e1d7abf65-istio-data\") pod \"e2e-ait-429de0b9-openshift-default-679c445bc7-2d549\" (UID: \"df056db4-2e39-44b9-8fa6-d50e1d7abf65\") " pod="openshift-ingress/e2e-ait-429de0b9-openshift-default-679c445bc7-2d549" Jul 13 22:38:10.103873 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:38:10.103685 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/df056db4-2e39-44b9-8fa6-d50e1d7abf65-workload-socket\") pod \"e2e-ait-429de0b9-openshift-default-679c445bc7-2d549\" (UID: \"df056db4-2e39-44b9-8fa6-d50e1d7abf65\") " pod="openshift-ingress/e2e-ait-429de0b9-openshift-default-679c445bc7-2d549" Jul 13 22:38:10.103873 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:38:10.103710 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/df056db4-2e39-44b9-8fa6-d50e1d7abf65-istio-envoy\") pod \"e2e-ait-429de0b9-openshift-default-679c445bc7-2d549\" (UID: \"df056db4-2e39-44b9-8fa6-d50e1d7abf65\") " pod="openshift-ingress/e2e-ait-429de0b9-openshift-default-679c445bc7-2d549" Jul 13 22:38:10.103873 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:38:10.103745 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/df056db4-2e39-44b9-8fa6-d50e1d7abf65-istio-podinfo\") pod \"e2e-ait-429de0b9-openshift-default-679c445bc7-2d549\" (UID: \"df056db4-2e39-44b9-8fa6-d50e1d7abf65\") " pod="openshift-ingress/e2e-ait-429de0b9-openshift-default-679c445bc7-2d549" Jul 13 22:38:10.103873 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:38:10.103782 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/df056db4-2e39-44b9-8fa6-d50e1d7abf65-istiod-ca-cert\") pod \"e2e-ait-429de0b9-openshift-default-679c445bc7-2d549\" (UID: \"df056db4-2e39-44b9-8fa6-d50e1d7abf65\") " pod="openshift-ingress/e2e-ait-429de0b9-openshift-default-679c445bc7-2d549" Jul 13 22:38:10.104164 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:38:10.103870 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/df056db4-2e39-44b9-8fa6-d50e1d7abf65-credential-socket\") pod \"e2e-ait-429de0b9-openshift-default-679c445bc7-2d549\" (UID: \"df056db4-2e39-44b9-8fa6-d50e1d7abf65\") " pod="openshift-ingress/e2e-ait-429de0b9-openshift-default-679c445bc7-2d549" Jul 13 22:38:10.104164 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:38:10.103903 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/df056db4-2e39-44b9-8fa6-d50e1d7abf65-istio-token\") pod \"e2e-ait-429de0b9-openshift-default-679c445bc7-2d549\" (UID: \"df056db4-2e39-44b9-8fa6-d50e1d7abf65\") " pod="openshift-ingress/e2e-ait-429de0b9-openshift-default-679c445bc7-2d549" Jul 13 22:38:10.104164 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:38:10.104089 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/df056db4-2e39-44b9-8fa6-d50e1d7abf65-workload-certs\") pod \"e2e-ait-429de0b9-openshift-default-679c445bc7-2d549\" (UID: \"df056db4-2e39-44b9-8fa6-d50e1d7abf65\") " pod="openshift-ingress/e2e-ait-429de0b9-openshift-default-679c445bc7-2d549" Jul 13 22:38:10.104164 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:38:10.104110 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/df056db4-2e39-44b9-8fa6-d50e1d7abf65-istio-data\") pod \"e2e-ait-429de0b9-openshift-default-679c445bc7-2d549\" (UID: \"df056db4-2e39-44b9-8fa6-d50e1d7abf65\") " pod="openshift-ingress/e2e-ait-429de0b9-openshift-default-679c445bc7-2d549" Jul 13 22:38:10.104164 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:38:10.104154 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/df056db4-2e39-44b9-8fa6-d50e1d7abf65-workload-socket\") pod \"e2e-ait-429de0b9-openshift-default-679c445bc7-2d549\" (UID: \"df056db4-2e39-44b9-8fa6-d50e1d7abf65\") " pod="openshift-ingress/e2e-ait-429de0b9-openshift-default-679c445bc7-2d549" Jul 13 22:38:10.104424 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:38:10.104192 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/df056db4-2e39-44b9-8fa6-d50e1d7abf65-credential-socket\") pod \"e2e-ait-429de0b9-openshift-default-679c445bc7-2d549\" (UID: \"df056db4-2e39-44b9-8fa6-d50e1d7abf65\") " pod="openshift-ingress/e2e-ait-429de0b9-openshift-default-679c445bc7-2d549" Jul 13 22:38:10.104701 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:38:10.104673 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/df056db4-2e39-44b9-8fa6-d50e1d7abf65-istiod-ca-cert\") pod \"e2e-ait-429de0b9-openshift-default-679c445bc7-2d549\" (UID: \"df056db4-2e39-44b9-8fa6-d50e1d7abf65\") " pod="openshift-ingress/e2e-ait-429de0b9-openshift-default-679c445bc7-2d549" Jul 13 22:38:10.106145 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:38:10.106118 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/df056db4-2e39-44b9-8fa6-d50e1d7abf65-istio-envoy\") pod \"e2e-ait-429de0b9-openshift-default-679c445bc7-2d549\" (UID: \"df056db4-2e39-44b9-8fa6-d50e1d7abf65\") " pod="openshift-ingress/e2e-ait-429de0b9-openshift-default-679c445bc7-2d549" Jul 13 22:38:10.106260 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:38:10.106242 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/df056db4-2e39-44b9-8fa6-d50e1d7abf65-istio-podinfo\") pod \"e2e-ait-429de0b9-openshift-default-679c445bc7-2d549\" (UID: \"df056db4-2e39-44b9-8fa6-d50e1d7abf65\") " pod="openshift-ingress/e2e-ait-429de0b9-openshift-default-679c445bc7-2d549" Jul 13 22:38:10.112967 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:38:10.112942 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/df056db4-2e39-44b9-8fa6-d50e1d7abf65-istio-token\") pod \"e2e-ait-429de0b9-openshift-default-679c445bc7-2d549\" (UID: \"df056db4-2e39-44b9-8fa6-d50e1d7abf65\") " pod="openshift-ingress/e2e-ait-429de0b9-openshift-default-679c445bc7-2d549" Jul 13 22:38:10.113104 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:38:10.113086 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-j8rll\" (UniqueName: \"kubernetes.io/projected/df056db4-2e39-44b9-8fa6-d50e1d7abf65-kube-api-access-j8rll\") pod \"e2e-ait-429de0b9-openshift-default-679c445bc7-2d549\" (UID: \"df056db4-2e39-44b9-8fa6-d50e1d7abf65\") " pod="openshift-ingress/e2e-ait-429de0b9-openshift-default-679c445bc7-2d549" Jul 13 22:38:10.223978 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:38:10.223878 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-429de0b9-openshift-default-679c445bc7-2d549" Jul 13 22:38:10.359918 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:38:10.359882 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-429de0b9-openshift-default-679c445bc7-2d549"] Jul 13 22:38:10.363213 ip-10-0-133-179 kubenswrapper[2583]: W0713 22:38:10.363183 2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddf056db4_2e39_44b9_8fa6_d50e1d7abf65.slice/crio-defbead905b4c62b3f29f6e535e231c6472466bf130422c3c2bd0f5209f1625b WatchSource:0}: Error finding container defbead905b4c62b3f29f6e535e231c6472466bf130422c3c2bd0f5209f1625b: Status 404 returned error can't find the container with id defbead905b4c62b3f29f6e535e231c6472466bf130422c3c2bd0f5209f1625b Jul 13 22:38:10.365137 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:38:10.365102 2583 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 13 22:38:10.365221 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:38:10.365171 2583 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 13 22:38:10.365221 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:38:10.365201 2583 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 13 22:38:11.192090 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:38:11.192051 2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-429de0b9-6d58c6b8db-mxt5j"] Jul 13 22:38:11.194734 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:38:11.194710 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-429de0b9-6d58c6b8db-mxt5j" Jul 13 22:38:11.197373 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:38:11.197350 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ait-429de0b9-serving-cert\"" Jul 13 22:38:11.206587 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:38:11.206536 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-429de0b9-6d58c6b8db-mxt5j"] Jul 13 22:38:11.207477 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:38:11.207445 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-429de0b9-openshift-default-679c445bc7-2d549" event={"ID":"df056db4-2e39-44b9-8fa6-d50e1d7abf65","Type":"ContainerStarted","Data":"7091ad21773b5d39c564f84ba3baa1cb8dc3c0ef6312aaa25ca3395f1249d47d"} Jul 13 22:38:11.207602 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:38:11.207499 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-429de0b9-openshift-default-679c445bc7-2d549" event={"ID":"df056db4-2e39-44b9-8fa6-d50e1d7abf65","Type":"ContainerStarted","Data":"defbead905b4c62b3f29f6e535e231c6472466bf130422c3c2bd0f5209f1625b"} Jul 13 22:38:11.224252 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:38:11.224217 2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-429de0b9-openshift-default-679c445bc7-2d549" Jul 13 22:38:11.229420 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:38:11.229398 2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-429de0b9-openshift-default-679c445bc7-2d549" Jul 13 22:38:11.250791 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:38:11.250742 2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-ait-429de0b9-openshift-default-679c445bc7-2d549" podStartSLOduration=2.250723659 podStartE2EDuration="2.250723659s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-13 22:38:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 22:38:11.24937047 +0000 UTC m=+1486.497285838" watchObservedRunningTime="2026-07-13 22:38:11.250723659 +0000 UTC m=+1486.498639009" Jul 13 22:38:11.316860 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:38:11.316826 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e89997fb-2a55-4f2d-9f62-557a99319907-maas-api-tls\") pod \"maas-api-e2e-ait-429de0b9-6d58c6b8db-mxt5j\" (UID: \"e89997fb-2a55-4f2d-9f62-557a99319907\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-429de0b9-6d58c6b8db-mxt5j" Jul 13 22:38:11.316860 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:38:11.316872 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e89997fb-2a55-4f2d-9f62-557a99319907-openshift-service-ca\") pod \"maas-api-e2e-ait-429de0b9-6d58c6b8db-mxt5j\" (UID: \"e89997fb-2a55-4f2d-9f62-557a99319907\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-429de0b9-6d58c6b8db-mxt5j" Jul 13 22:38:11.317148 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:38:11.317110 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d5tmj\" (UniqueName: \"kubernetes.io/projected/e89997fb-2a55-4f2d-9f62-557a99319907-kube-api-access-d5tmj\") pod \"maas-api-e2e-ait-429de0b9-6d58c6b8db-mxt5j\" (UID: \"e89997fb-2a55-4f2d-9f62-557a99319907\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-429de0b9-6d58c6b8db-mxt5j" Jul 13 22:38:11.390240 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:38:11.390205 2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="db37f4c7-fff5-491d-b3c4-b8f33ab5b2da" path="/var/lib/kubelet/pods/db37f4c7-fff5-491d-b3c4-b8f33ab5b2da/volumes" Jul 13 22:38:11.417653 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:38:11.417608 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d5tmj\" (UniqueName: \"kubernetes.io/projected/e89997fb-2a55-4f2d-9f62-557a99319907-kube-api-access-d5tmj\") pod \"maas-api-e2e-ait-429de0b9-6d58c6b8db-mxt5j\" (UID: \"e89997fb-2a55-4f2d-9f62-557a99319907\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-429de0b9-6d58c6b8db-mxt5j" Jul 13 22:38:11.417859 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:38:11.417668 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e89997fb-2a55-4f2d-9f62-557a99319907-maas-api-tls\") pod \"maas-api-e2e-ait-429de0b9-6d58c6b8db-mxt5j\" (UID: \"e89997fb-2a55-4f2d-9f62-557a99319907\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-429de0b9-6d58c6b8db-mxt5j" Jul 13 22:38:11.417859 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:38:11.417714 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e89997fb-2a55-4f2d-9f62-557a99319907-openshift-service-ca\") pod \"maas-api-e2e-ait-429de0b9-6d58c6b8db-mxt5j\" (UID: \"e89997fb-2a55-4f2d-9f62-557a99319907\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-429de0b9-6d58c6b8db-mxt5j" Jul 13 22:38:11.418373 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:38:11.418347 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e89997fb-2a55-4f2d-9f62-557a99319907-openshift-service-ca\") pod \"maas-api-e2e-ait-429de0b9-6d58c6b8db-mxt5j\" (UID: \"e89997fb-2a55-4f2d-9f62-557a99319907\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-429de0b9-6d58c6b8db-mxt5j" Jul 13 22:38:11.420213 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:38:11.420190 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e89997fb-2a55-4f2d-9f62-557a99319907-maas-api-tls\") pod \"maas-api-e2e-ait-429de0b9-6d58c6b8db-mxt5j\" (UID: \"e89997fb-2a55-4f2d-9f62-557a99319907\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-429de0b9-6d58c6b8db-mxt5j" Jul 13 22:38:11.439180 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:38:11.439140 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-d5tmj\" (UniqueName: \"kubernetes.io/projected/e89997fb-2a55-4f2d-9f62-557a99319907-kube-api-access-d5tmj\") pod \"maas-api-e2e-ait-429de0b9-6d58c6b8db-mxt5j\" (UID: \"e89997fb-2a55-4f2d-9f62-557a99319907\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-429de0b9-6d58c6b8db-mxt5j" Jul 13 22:38:11.508004 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:38:11.507914 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-429de0b9-6d58c6b8db-mxt5j" Jul 13 22:38:11.651525 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:38:11.651499 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-429de0b9-6d58c6b8db-mxt5j"] Jul 13 22:38:11.652745 ip-10-0-133-179 kubenswrapper[2583]: W0713 22:38:11.652713 2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode89997fb_2a55_4f2d_9f62_557a99319907.slice/crio-a675c3d8e8e82e3fec15cbfe3ab533f8d0c3daac64339f6c4f90538a4f076d84 WatchSource:0}: Error finding container a675c3d8e8e82e3fec15cbfe3ab533f8d0c3daac64339f6c4f90538a4f076d84: Status 404 returned error can't find the container with id a675c3d8e8e82e3fec15cbfe3ab533f8d0c3daac64339f6c4f90538a4f076d84 Jul 13 22:38:12.213912 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:38:12.213869 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-429de0b9-6d58c6b8db-mxt5j" event={"ID":"e89997fb-2a55-4f2d-9f62-557a99319907","Type":"ContainerStarted","Data":"9cf90076415ccb68ce279d42d0fd7ca4a6cc0c290cabd67e64e131c5350e3af6"} Jul 13 22:38:12.214109 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:38:12.213926 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-429de0b9-6d58c6b8db-mxt5j" event={"ID":"e89997fb-2a55-4f2d-9f62-557a99319907","Type":"ContainerStarted","Data":"a675c3d8e8e82e3fec15cbfe3ab533f8d0c3daac64339f6c4f90538a4f076d84"} Jul 13 22:38:12.214176 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:38:12.214161 2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-429de0b9-openshift-default-679c445bc7-2d549" Jul 13 22:38:12.214230 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:38:12.214207 2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-429de0b9-6d58c6b8db-mxt5j" Jul 13 22:38:12.215218 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:38:12.215200 2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-429de0b9-openshift-default-679c445bc7-2d549" Jul 13 22:38:12.235865 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:38:12.235810 2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ait-429de0b9-6d58c6b8db-mxt5j" podStartSLOduration=0.854324843 podStartE2EDuration="1.235793757s" totalImagesPullingTime="381.468914ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-13 22:38:11 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 22:38:12.234796384 +0000 UTC m=+1487.482711738" watchObservedRunningTime="2026-07-13 22:38:12.235793757 +0000 UTC m=+1487.483709107" Jul 13 22:38:16.496540 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:38:16.496498 2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-429de0b9-6d58c6b8db-mxt5j"] Jul 13 22:38:16.497040 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:38:16.496820 2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ait-429de0b9-6d58c6b8db-mxt5j" podUID="e89997fb-2a55-4f2d-9f62-557a99319907" containerName="maas-api" containerID="cri-o://9cf90076415ccb68ce279d42d0fd7ca4a6cc0c290cabd67e64e131c5350e3af6" gracePeriod=30 Jul 13 22:38:16.742461 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:38:16.742426 2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-429de0b9-6d58c6b8db-mxt5j" Jul 13 22:38:16.764340 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:38:16.764266 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e89997fb-2a55-4f2d-9f62-557a99319907-openshift-service-ca\") pod \"e89997fb-2a55-4f2d-9f62-557a99319907\" (UID: \"e89997fb-2a55-4f2d-9f62-557a99319907\") " Jul 13 22:38:16.764340 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:38:16.764300 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-d5tmj\" (UniqueName: \"kubernetes.io/projected/e89997fb-2a55-4f2d-9f62-557a99319907-kube-api-access-d5tmj\") pod \"e89997fb-2a55-4f2d-9f62-557a99319907\" (UID: \"e89997fb-2a55-4f2d-9f62-557a99319907\") " Jul 13 22:38:16.764340 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:38:16.764336 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e89997fb-2a55-4f2d-9f62-557a99319907-maas-api-tls\") pod \"e89997fb-2a55-4f2d-9f62-557a99319907\" (UID: \"e89997fb-2a55-4f2d-9f62-557a99319907\") " Jul 13 22:38:16.764748 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:38:16.764718 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e89997fb-2a55-4f2d-9f62-557a99319907-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "e89997fb-2a55-4f2d-9f62-557a99319907" (UID: "e89997fb-2a55-4f2d-9f62-557a99319907"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 13 22:38:16.766517 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:38:16.766489 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e89997fb-2a55-4f2d-9f62-557a99319907-kube-api-access-d5tmj" (OuterVolumeSpecName: "kube-api-access-d5tmj") pod "e89997fb-2a55-4f2d-9f62-557a99319907" (UID: "e89997fb-2a55-4f2d-9f62-557a99319907"). InnerVolumeSpecName "kube-api-access-d5tmj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 13 22:38:16.766517 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:38:16.766508 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e89997fb-2a55-4f2d-9f62-557a99319907-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "e89997fb-2a55-4f2d-9f62-557a99319907" (UID: "e89997fb-2a55-4f2d-9f62-557a99319907"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 13 22:38:16.865126 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:38:16.865092 2583 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e89997fb-2a55-4f2d-9f62-557a99319907-maas-api-tls\") on node \"ip-10-0-133-179.ec2.internal\" DevicePath \"\"" Jul 13 22:38:16.865126 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:38:16.865125 2583 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e89997fb-2a55-4f2d-9f62-557a99319907-openshift-service-ca\") on node \"ip-10-0-133-179.ec2.internal\" DevicePath \"\"" Jul 13 22:38:16.865327 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:38:16.865138 2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d5tmj\" (UniqueName: \"kubernetes.io/projected/e89997fb-2a55-4f2d-9f62-557a99319907-kube-api-access-d5tmj\") on node \"ip-10-0-133-179.ec2.internal\" DevicePath \"\"" Jul 13 22:38:17.243528 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:38:17.243493 2583 generic.go:356] "Generic (PLEG): container finished" podID="e89997fb-2a55-4f2d-9f62-557a99319907" containerID="9cf90076415ccb68ce279d42d0fd7ca4a6cc0c290cabd67e64e131c5350e3af6" exitCode=0 Jul 13 22:38:17.243741 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:38:17.243537 2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-429de0b9-6d58c6b8db-mxt5j" Jul 13 22:38:17.243741 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:38:17.243543 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-429de0b9-6d58c6b8db-mxt5j" event={"ID":"e89997fb-2a55-4f2d-9f62-557a99319907","Type":"ContainerDied","Data":"9cf90076415ccb68ce279d42d0fd7ca4a6cc0c290cabd67e64e131c5350e3af6"} Jul 13 22:38:17.243741 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:38:17.243614 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-429de0b9-6d58c6b8db-mxt5j" event={"ID":"e89997fb-2a55-4f2d-9f62-557a99319907","Type":"ContainerDied","Data":"a675c3d8e8e82e3fec15cbfe3ab533f8d0c3daac64339f6c4f90538a4f076d84"} Jul 13 22:38:17.243741 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:38:17.243638 2583 scope.go:117] "RemoveContainer" containerID="9cf90076415ccb68ce279d42d0fd7ca4a6cc0c290cabd67e64e131c5350e3af6" Jul 13 22:38:17.252333 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:38:17.252312 2583 scope.go:117] "RemoveContainer" containerID="9cf90076415ccb68ce279d42d0fd7ca4a6cc0c290cabd67e64e131c5350e3af6" Jul 13 22:38:17.252557 ip-10-0-133-179 kubenswrapper[2583]: E0713 22:38:17.252536 2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9cf90076415ccb68ce279d42d0fd7ca4a6cc0c290cabd67e64e131c5350e3af6\": container with ID starting with 9cf90076415ccb68ce279d42d0fd7ca4a6cc0c290cabd67e64e131c5350e3af6 not found: ID does not exist" containerID="9cf90076415ccb68ce279d42d0fd7ca4a6cc0c290cabd67e64e131c5350e3af6" Jul 13 22:38:17.252644 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:38:17.252590 2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9cf90076415ccb68ce279d42d0fd7ca4a6cc0c290cabd67e64e131c5350e3af6"} err="failed to get container status \"9cf90076415ccb68ce279d42d0fd7ca4a6cc0c290cabd67e64e131c5350e3af6\": rpc error: code = NotFound desc = could not find container \"9cf90076415ccb68ce279d42d0fd7ca4a6cc0c290cabd67e64e131c5350e3af6\": container with ID starting with 9cf90076415ccb68ce279d42d0fd7ca4a6cc0c290cabd67e64e131c5350e3af6 not found: ID does not exist" Jul 13 22:38:17.265612 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:38:17.265556 2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-429de0b9-6d58c6b8db-mxt5j"] Jul 13 22:38:17.268163 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:38:17.268137 2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-429de0b9-6d58c6b8db-mxt5j"] Jul 13 22:38:17.389810 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:38:17.389774 2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e89997fb-2a55-4f2d-9f62-557a99319907" path="/var/lib/kubelet/pods/e89997fb-2a55-4f2d-9f62-557a99319907/volumes" Jul 13 22:38:22.248589 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:38:22.248539 2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-429de0b9-openshift-default-679c445bc7-2d549"] Jul 13 22:38:22.248996 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:38:22.248758 2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-ait-429de0b9-openshift-default-679c445bc7-2d549" podUID="df056db4-2e39-44b9-8fa6-d50e1d7abf65" containerName="istio-proxy" containerID="cri-o://7091ad21773b5d39c564f84ba3baa1cb8dc3c0ef6312aaa25ca3395f1249d47d" gracePeriod=30 Jul 13 22:38:27.217130 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:38:27.217092 2583 patch_prober.go:28] interesting pod/e2e-ait-429de0b9-openshift-default-679c445bc7-2d549 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Jul 13 22:38:27.217530 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:38:27.217167 2583 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ait-429de0b9-openshift-default-679c445bc7-2d549" podUID="df056db4-2e39-44b9-8fa6-d50e1d7abf65" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 13 22:38:27.497472 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:38:27.497445 2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-429de0b9-openshift-default-679c445bc7-2d549" Jul 13 22:38:27.558717 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:38:27.558684 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-j8rll\" (UniqueName: \"kubernetes.io/projected/df056db4-2e39-44b9-8fa6-d50e1d7abf65-kube-api-access-j8rll\") pod \"df056db4-2e39-44b9-8fa6-d50e1d7abf65\" (UID: \"df056db4-2e39-44b9-8fa6-d50e1d7abf65\") " Jul 13 22:38:27.558942 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:38:27.558744 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/df056db4-2e39-44b9-8fa6-d50e1d7abf65-istiod-ca-cert\") pod \"df056db4-2e39-44b9-8fa6-d50e1d7abf65\" (UID: \"df056db4-2e39-44b9-8fa6-d50e1d7abf65\") " Jul 13 22:38:27.558942 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:38:27.558767 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/df056db4-2e39-44b9-8fa6-d50e1d7abf65-workload-certs\") pod \"df056db4-2e39-44b9-8fa6-d50e1d7abf65\" (UID: \"df056db4-2e39-44b9-8fa6-d50e1d7abf65\") " Jul 13 22:38:27.558942 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:38:27.558800 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/df056db4-2e39-44b9-8fa6-d50e1d7abf65-workload-socket\") pod \"df056db4-2e39-44b9-8fa6-d50e1d7abf65\" (UID: \"df056db4-2e39-44b9-8fa6-d50e1d7abf65\") " Jul 13 22:38:27.558942 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:38:27.558871 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/df056db4-2e39-44b9-8fa6-d50e1d7abf65-istio-podinfo\") pod \"df056db4-2e39-44b9-8fa6-d50e1d7abf65\" (UID: \"df056db4-2e39-44b9-8fa6-d50e1d7abf65\") " Jul 13 22:38:27.558942 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:38:27.558915 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/df056db4-2e39-44b9-8fa6-d50e1d7abf65-istio-token\") pod \"df056db4-2e39-44b9-8fa6-d50e1d7abf65\" (UID: \"df056db4-2e39-44b9-8fa6-d50e1d7abf65\") " Jul 13 22:38:27.558942 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:38:27.558941 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/df056db4-2e39-44b9-8fa6-d50e1d7abf65-istio-data\") pod \"df056db4-2e39-44b9-8fa6-d50e1d7abf65\" (UID: \"df056db4-2e39-44b9-8fa6-d50e1d7abf65\") " Jul 13 22:38:27.559231 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:38:27.558974 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/df056db4-2e39-44b9-8fa6-d50e1d7abf65-credential-socket\") pod \"df056db4-2e39-44b9-8fa6-d50e1d7abf65\" (UID: \"df056db4-2e39-44b9-8fa6-d50e1d7abf65\") " Jul 13 22:38:27.559231 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:38:27.559012 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/df056db4-2e39-44b9-8fa6-d50e1d7abf65-istio-envoy\") pod \"df056db4-2e39-44b9-8fa6-d50e1d7abf65\" (UID: \"df056db4-2e39-44b9-8fa6-d50e1d7abf65\") " Jul 13 22:38:27.559231 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:38:27.559166 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/df056db4-2e39-44b9-8fa6-d50e1d7abf65-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "df056db4-2e39-44b9-8fa6-d50e1d7abf65" (UID: "df056db4-2e39-44b9-8fa6-d50e1d7abf65"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 13 22:38:27.559231 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:38:27.559196 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/df056db4-2e39-44b9-8fa6-d50e1d7abf65-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "df056db4-2e39-44b9-8fa6-d50e1d7abf65" (UID: "df056db4-2e39-44b9-8fa6-d50e1d7abf65"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 13 22:38:27.559431 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:38:27.559295 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/df056db4-2e39-44b9-8fa6-d50e1d7abf65-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "df056db4-2e39-44b9-8fa6-d50e1d7abf65" (UID: "df056db4-2e39-44b9-8fa6-d50e1d7abf65"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 13 22:38:27.559431 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:38:27.559314 2583 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/df056db4-2e39-44b9-8fa6-d50e1d7abf65-workload-certs\") on node \"ip-10-0-133-179.ec2.internal\" DevicePath \"\"" Jul 13 22:38:27.559431 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:38:27.559333 2583 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/df056db4-2e39-44b9-8fa6-d50e1d7abf65-workload-socket\") on node \"ip-10-0-133-179.ec2.internal\" DevicePath \"\"" Jul 13 22:38:27.559431 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:38:27.559408 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/df056db4-2e39-44b9-8fa6-d50e1d7abf65-istio-data" (OuterVolumeSpecName: "istio-data") pod "df056db4-2e39-44b9-8fa6-d50e1d7abf65" (UID: "df056db4-2e39-44b9-8fa6-d50e1d7abf65"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 13 22:38:27.559782 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:38:27.559756 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/df056db4-2e39-44b9-8fa6-d50e1d7abf65-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "df056db4-2e39-44b9-8fa6-d50e1d7abf65" (UID: "df056db4-2e39-44b9-8fa6-d50e1d7abf65"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 13 22:38:27.561764 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:38:27.561738 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/df056db4-2e39-44b9-8fa6-d50e1d7abf65-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "df056db4-2e39-44b9-8fa6-d50e1d7abf65" (UID: "df056db4-2e39-44b9-8fa6-d50e1d7abf65"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 13 22:38:27.561909 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:38:27.561884 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/df056db4-2e39-44b9-8fa6-d50e1d7abf65-istio-token" (OuterVolumeSpecName: "istio-token") pod "df056db4-2e39-44b9-8fa6-d50e1d7abf65" (UID: "df056db4-2e39-44b9-8fa6-d50e1d7abf65"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 13 22:38:27.562096 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:38:27.562073 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/df056db4-2e39-44b9-8fa6-d50e1d7abf65-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "df056db4-2e39-44b9-8fa6-d50e1d7abf65" (UID: "df056db4-2e39-44b9-8fa6-d50e1d7abf65"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 13 22:38:27.562172 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:38:27.562106 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/df056db4-2e39-44b9-8fa6-d50e1d7abf65-kube-api-access-j8rll" (OuterVolumeSpecName: "kube-api-access-j8rll") pod "df056db4-2e39-44b9-8fa6-d50e1d7abf65" (UID: "df056db4-2e39-44b9-8fa6-d50e1d7abf65"). InnerVolumeSpecName "kube-api-access-j8rll". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 13 22:38:27.659920 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:38:27.659867 2583 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/df056db4-2e39-44b9-8fa6-d50e1d7abf65-istio-podinfo\") on node \"ip-10-0-133-179.ec2.internal\" DevicePath \"\"" Jul 13 22:38:27.659920 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:38:27.659915 2583 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/df056db4-2e39-44b9-8fa6-d50e1d7abf65-istio-token\") on node \"ip-10-0-133-179.ec2.internal\" DevicePath \"\"" Jul 13 22:38:27.659920 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:38:27.659926 2583 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/df056db4-2e39-44b9-8fa6-d50e1d7abf65-istio-data\") on node \"ip-10-0-133-179.ec2.internal\" DevicePath \"\"" Jul 13 22:38:27.659920 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:38:27.659937 2583 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/df056db4-2e39-44b9-8fa6-d50e1d7abf65-credential-socket\") on node \"ip-10-0-133-179.ec2.internal\" DevicePath \"\"" Jul 13 22:38:27.660190 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:38:27.659945 2583 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/df056db4-2e39-44b9-8fa6-d50e1d7abf65-istio-envoy\") on node \"ip-10-0-133-179.ec2.internal\" DevicePath \"\"" Jul 13 22:38:27.660190 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:38:27.659954 2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j8rll\" (UniqueName: \"kubernetes.io/projected/df056db4-2e39-44b9-8fa6-d50e1d7abf65-kube-api-access-j8rll\") on node \"ip-10-0-133-179.ec2.internal\" DevicePath \"\"" Jul 13 22:38:27.660190 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:38:27.659963 2583 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/df056db4-2e39-44b9-8fa6-d50e1d7abf65-istiod-ca-cert\") on node \"ip-10-0-133-179.ec2.internal\" DevicePath \"\"" Jul 13 22:38:28.291422 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:38:28.291380 2583 generic.go:356] "Generic (PLEG): container finished" podID="df056db4-2e39-44b9-8fa6-d50e1d7abf65" containerID="7091ad21773b5d39c564f84ba3baa1cb8dc3c0ef6312aaa25ca3395f1249d47d" exitCode=0 Jul 13 22:38:28.291870 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:38:28.291433 2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-429de0b9-openshift-default-679c445bc7-2d549" Jul 13 22:38:28.291870 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:38:28.291465 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-429de0b9-openshift-default-679c445bc7-2d549" event={"ID":"df056db4-2e39-44b9-8fa6-d50e1d7abf65","Type":"ContainerDied","Data":"7091ad21773b5d39c564f84ba3baa1cb8dc3c0ef6312aaa25ca3395f1249d47d"} Jul 13 22:38:28.291870 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:38:28.291523 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-429de0b9-openshift-default-679c445bc7-2d549" event={"ID":"df056db4-2e39-44b9-8fa6-d50e1d7abf65","Type":"ContainerDied","Data":"defbead905b4c62b3f29f6e535e231c6472466bf130422c3c2bd0f5209f1625b"} Jul 13 22:38:28.291870 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:38:28.291547 2583 scope.go:117] "RemoveContainer" containerID="7091ad21773b5d39c564f84ba3baa1cb8dc3c0ef6312aaa25ca3395f1249d47d" Jul 13 22:38:28.300966 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:38:28.300939 2583 scope.go:117] "RemoveContainer" containerID="7091ad21773b5d39c564f84ba3baa1cb8dc3c0ef6312aaa25ca3395f1249d47d" Jul 13 22:38:28.301355 ip-10-0-133-179 kubenswrapper[2583]: E0713 22:38:28.301330 2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7091ad21773b5d39c564f84ba3baa1cb8dc3c0ef6312aaa25ca3395f1249d47d\": container with ID starting with 7091ad21773b5d39c564f84ba3baa1cb8dc3c0ef6312aaa25ca3395f1249d47d not found: ID does not exist" containerID="7091ad21773b5d39c564f84ba3baa1cb8dc3c0ef6312aaa25ca3395f1249d47d" Jul 13 22:38:28.301451 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:38:28.301364 2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7091ad21773b5d39c564f84ba3baa1cb8dc3c0ef6312aaa25ca3395f1249d47d"} err="failed to get container status \"7091ad21773b5d39c564f84ba3baa1cb8dc3c0ef6312aaa25ca3395f1249d47d\": rpc error: code = NotFound desc = could not find container \"7091ad21773b5d39c564f84ba3baa1cb8dc3c0ef6312aaa25ca3395f1249d47d\": container with ID starting with 7091ad21773b5d39c564f84ba3baa1cb8dc3c0ef6312aaa25ca3395f1249d47d not found: ID does not exist" Jul 13 22:38:28.319351 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:38:28.319318 2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-429de0b9-openshift-default-679c445bc7-2d549"] Jul 13 22:38:28.327759 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:38:28.327722 2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ait-429de0b9-openshift-default-679c445bc7-2d549"] Jul 13 22:38:28.342379 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:38:28.342346 2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-derive-036ddcf7-openshift-default-86c6b5fc8c-mznzj"] Jul 13 22:38:28.342827 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:38:28.342806 2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e89997fb-2a55-4f2d-9f62-557a99319907" containerName="maas-api" Jul 13 22:38:28.342827 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:38:28.342828 2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="e89997fb-2a55-4f2d-9f62-557a99319907" containerName="maas-api" Jul 13 22:38:28.342975 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:38:28.342845 2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="df056db4-2e39-44b9-8fa6-d50e1d7abf65" containerName="istio-proxy" Jul 13 22:38:28.342975 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:38:28.342853 2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="df056db4-2e39-44b9-8fa6-d50e1d7abf65" containerName="istio-proxy" Jul 13 22:38:28.342975 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:38:28.342937 2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="df056db4-2e39-44b9-8fa6-d50e1d7abf65" containerName="istio-proxy" Jul 13 22:38:28.342975 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:38:28.342951 2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="e89997fb-2a55-4f2d-9f62-557a99319907" containerName="maas-api" Jul 13 22:38:28.346192 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:38:28.346168 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-036ddcf7-openshift-default-86c6b5fc8c-mznzj" Jul 13 22:38:28.348543 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:38:28.348515 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-derive-036ddcf7-openshift-default-dockercfg-mcszb\"" Jul 13 22:38:28.360692 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:38:28.360667 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-derive-036ddcf7-openshift-default-86c6b5fc8c-mznzj"] Jul 13 22:38:28.468626 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:38:28.468583 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4a0e4a31-336c-4ac7-8ad8-9b2ea689099c-istio-envoy\") pod \"e2e-derive-036ddcf7-openshift-default-86c6b5fc8c-mznzj\" (UID: \"4a0e4a31-336c-4ac7-8ad8-9b2ea689099c\") " pod="openshift-ingress/e2e-derive-036ddcf7-openshift-default-86c6b5fc8c-mznzj" Jul 13 22:38:28.468626 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:38:28.468633 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4a0e4a31-336c-4ac7-8ad8-9b2ea689099c-workload-socket\") pod \"e2e-derive-036ddcf7-openshift-default-86c6b5fc8c-mznzj\" (UID: \"4a0e4a31-336c-4ac7-8ad8-9b2ea689099c\") " pod="openshift-ingress/e2e-derive-036ddcf7-openshift-default-86c6b5fc8c-mznzj" Jul 13 22:38:28.468891 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:38:28.468656 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4a0e4a31-336c-4ac7-8ad8-9b2ea689099c-workload-certs\") pod \"e2e-derive-036ddcf7-openshift-default-86c6b5fc8c-mznzj\" (UID: \"4a0e4a31-336c-4ac7-8ad8-9b2ea689099c\") " pod="openshift-ingress/e2e-derive-036ddcf7-openshift-default-86c6b5fc8c-mznzj" Jul 13 22:38:28.468891 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:38:28.468674 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5f7bf\" (UniqueName: \"kubernetes.io/projected/4a0e4a31-336c-4ac7-8ad8-9b2ea689099c-kube-api-access-5f7bf\") pod \"e2e-derive-036ddcf7-openshift-default-86c6b5fc8c-mznzj\" (UID: \"4a0e4a31-336c-4ac7-8ad8-9b2ea689099c\") " pod="openshift-ingress/e2e-derive-036ddcf7-openshift-default-86c6b5fc8c-mznzj" Jul 13 22:38:28.468891 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:38:28.468691 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4a0e4a31-336c-4ac7-8ad8-9b2ea689099c-istio-token\") pod \"e2e-derive-036ddcf7-openshift-default-86c6b5fc8c-mznzj\" (UID: \"4a0e4a31-336c-4ac7-8ad8-9b2ea689099c\") " pod="openshift-ingress/e2e-derive-036ddcf7-openshift-default-86c6b5fc8c-mznzj" Jul 13 22:38:28.468891 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:38:28.468794 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4a0e4a31-336c-4ac7-8ad8-9b2ea689099c-credential-socket\") pod \"e2e-derive-036ddcf7-openshift-default-86c6b5fc8c-mznzj\" (UID: \"4a0e4a31-336c-4ac7-8ad8-9b2ea689099c\") " pod="openshift-ingress/e2e-derive-036ddcf7-openshift-default-86c6b5fc8c-mznzj" Jul 13 22:38:28.468891 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:38:28.468837 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4a0e4a31-336c-4ac7-8ad8-9b2ea689099c-istio-data\") pod \"e2e-derive-036ddcf7-openshift-default-86c6b5fc8c-mznzj\" (UID: \"4a0e4a31-336c-4ac7-8ad8-9b2ea689099c\") " pod="openshift-ingress/e2e-derive-036ddcf7-openshift-default-86c6b5fc8c-mznzj" Jul 13 22:38:28.468891 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:38:28.468856 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4a0e4a31-336c-4ac7-8ad8-9b2ea689099c-istio-podinfo\") pod \"e2e-derive-036ddcf7-openshift-default-86c6b5fc8c-mznzj\" (UID: \"4a0e4a31-336c-4ac7-8ad8-9b2ea689099c\") " pod="openshift-ingress/e2e-derive-036ddcf7-openshift-default-86c6b5fc8c-mznzj" Jul 13 22:38:28.469192 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:38:28.468914 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4a0e4a31-336c-4ac7-8ad8-9b2ea689099c-istiod-ca-cert\") pod \"e2e-derive-036ddcf7-openshift-default-86c6b5fc8c-mznzj\" (UID: \"4a0e4a31-336c-4ac7-8ad8-9b2ea689099c\") " pod="openshift-ingress/e2e-derive-036ddcf7-openshift-default-86c6b5fc8c-mznzj" Jul 13 22:38:28.570343 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:38:28.570252 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4a0e4a31-336c-4ac7-8ad8-9b2ea689099c-istio-envoy\") pod \"e2e-derive-036ddcf7-openshift-default-86c6b5fc8c-mznzj\" (UID: \"4a0e4a31-336c-4ac7-8ad8-9b2ea689099c\") " pod="openshift-ingress/e2e-derive-036ddcf7-openshift-default-86c6b5fc8c-mznzj" Jul 13 22:38:28.570343 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:38:28.570292 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4a0e4a31-336c-4ac7-8ad8-9b2ea689099c-workload-socket\") pod \"e2e-derive-036ddcf7-openshift-default-86c6b5fc8c-mznzj\" (UID: \"4a0e4a31-336c-4ac7-8ad8-9b2ea689099c\") " pod="openshift-ingress/e2e-derive-036ddcf7-openshift-default-86c6b5fc8c-mznzj" Jul 13 22:38:28.570343 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:38:28.570310 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4a0e4a31-336c-4ac7-8ad8-9b2ea689099c-workload-certs\") pod \"e2e-derive-036ddcf7-openshift-default-86c6b5fc8c-mznzj\" (UID: \"4a0e4a31-336c-4ac7-8ad8-9b2ea689099c\") " pod="openshift-ingress/e2e-derive-036ddcf7-openshift-default-86c6b5fc8c-mznzj" Jul 13 22:38:28.570620 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:38:28.570434 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5f7bf\" (UniqueName: \"kubernetes.io/projected/4a0e4a31-336c-4ac7-8ad8-9b2ea689099c-kube-api-access-5f7bf\") pod \"e2e-derive-036ddcf7-openshift-default-86c6b5fc8c-mznzj\" (UID: \"4a0e4a31-336c-4ac7-8ad8-9b2ea689099c\") " pod="openshift-ingress/e2e-derive-036ddcf7-openshift-default-86c6b5fc8c-mznzj" Jul 13 22:38:28.570620 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:38:28.570479 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4a0e4a31-336c-4ac7-8ad8-9b2ea689099c-istio-token\") pod \"e2e-derive-036ddcf7-openshift-default-86c6b5fc8c-mznzj\" (UID: \"4a0e4a31-336c-4ac7-8ad8-9b2ea689099c\") " pod="openshift-ingress/e2e-derive-036ddcf7-openshift-default-86c6b5fc8c-mznzj" Jul 13 22:38:28.570620 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:38:28.570584 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4a0e4a31-336c-4ac7-8ad8-9b2ea689099c-credential-socket\") pod \"e2e-derive-036ddcf7-openshift-default-86c6b5fc8c-mznzj\" (UID: \"4a0e4a31-336c-4ac7-8ad8-9b2ea689099c\") " pod="openshift-ingress/e2e-derive-036ddcf7-openshift-default-86c6b5fc8c-mznzj" Jul 13 22:38:28.570760 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:38:28.570636 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4a0e4a31-336c-4ac7-8ad8-9b2ea689099c-istio-data\") pod \"e2e-derive-036ddcf7-openshift-default-86c6b5fc8c-mznzj\" (UID: \"4a0e4a31-336c-4ac7-8ad8-9b2ea689099c\") " pod="openshift-ingress/e2e-derive-036ddcf7-openshift-default-86c6b5fc8c-mznzj" Jul 13 22:38:28.570760 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:38:28.570652 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4a0e4a31-336c-4ac7-8ad8-9b2ea689099c-workload-socket\") pod \"e2e-derive-036ddcf7-openshift-default-86c6b5fc8c-mznzj\" (UID: \"4a0e4a31-336c-4ac7-8ad8-9b2ea689099c\") " pod="openshift-ingress/e2e-derive-036ddcf7-openshift-default-86c6b5fc8c-mznzj" Jul 13 22:38:28.570760 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:38:28.570660 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4a0e4a31-336c-4ac7-8ad8-9b2ea689099c-istio-podinfo\") pod \"e2e-derive-036ddcf7-openshift-default-86c6b5fc8c-mznzj\" (UID: \"4a0e4a31-336c-4ac7-8ad8-9b2ea689099c\") " pod="openshift-ingress/e2e-derive-036ddcf7-openshift-default-86c6b5fc8c-mznzj" Jul 13 22:38:28.570760 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:38:28.570718 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4a0e4a31-336c-4ac7-8ad8-9b2ea689099c-istiod-ca-cert\") pod \"e2e-derive-036ddcf7-openshift-default-86c6b5fc8c-mznzj\" (UID: \"4a0e4a31-336c-4ac7-8ad8-9b2ea689099c\") " pod="openshift-ingress/e2e-derive-036ddcf7-openshift-default-86c6b5fc8c-mznzj" Jul 13 22:38:28.570932 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:38:28.570774 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4a0e4a31-336c-4ac7-8ad8-9b2ea689099c-workload-certs\") pod \"e2e-derive-036ddcf7-openshift-default-86c6b5fc8c-mznzj\" (UID: \"4a0e4a31-336c-4ac7-8ad8-9b2ea689099c\") " pod="openshift-ingress/e2e-derive-036ddcf7-openshift-default-86c6b5fc8c-mznzj" Jul 13 22:38:28.570932 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:38:28.570908 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4a0e4a31-336c-4ac7-8ad8-9b2ea689099c-credential-socket\") pod \"e2e-derive-036ddcf7-openshift-default-86c6b5fc8c-mznzj\" (UID: \"4a0e4a31-336c-4ac7-8ad8-9b2ea689099c\") " pod="openshift-ingress/e2e-derive-036ddcf7-openshift-default-86c6b5fc8c-mznzj" Jul 13 22:38:28.570994 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:38:28.570967 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4a0e4a31-336c-4ac7-8ad8-9b2ea689099c-istio-data\") pod \"e2e-derive-036ddcf7-openshift-default-86c6b5fc8c-mznzj\" (UID: \"4a0e4a31-336c-4ac7-8ad8-9b2ea689099c\") " pod="openshift-ingress/e2e-derive-036ddcf7-openshift-default-86c6b5fc8c-mznzj" Jul 13 22:38:28.571372 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:38:28.571351 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4a0e4a31-336c-4ac7-8ad8-9b2ea689099c-istiod-ca-cert\") pod \"e2e-derive-036ddcf7-openshift-default-86c6b5fc8c-mznzj\" (UID: \"4a0e4a31-336c-4ac7-8ad8-9b2ea689099c\") " pod="openshift-ingress/e2e-derive-036ddcf7-openshift-default-86c6b5fc8c-mznzj" Jul 13 22:38:28.572591 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:38:28.572550 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4a0e4a31-336c-4ac7-8ad8-9b2ea689099c-istio-envoy\") pod \"e2e-derive-036ddcf7-openshift-default-86c6b5fc8c-mznzj\" (UID: \"4a0e4a31-336c-4ac7-8ad8-9b2ea689099c\") " pod="openshift-ingress/e2e-derive-036ddcf7-openshift-default-86c6b5fc8c-mznzj" Jul 13 22:38:28.573097 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:38:28.573078 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4a0e4a31-336c-4ac7-8ad8-9b2ea689099c-istio-podinfo\") pod \"e2e-derive-036ddcf7-openshift-default-86c6b5fc8c-mznzj\" (UID: \"4a0e4a31-336c-4ac7-8ad8-9b2ea689099c\") " pod="openshift-ingress/e2e-derive-036ddcf7-openshift-default-86c6b5fc8c-mznzj" Jul 13 22:38:28.578484 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:38:28.578462 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4a0e4a31-336c-4ac7-8ad8-9b2ea689099c-istio-token\") pod \"e2e-derive-036ddcf7-openshift-default-86c6b5fc8c-mznzj\" (UID: \"4a0e4a31-336c-4ac7-8ad8-9b2ea689099c\") " pod="openshift-ingress/e2e-derive-036ddcf7-openshift-default-86c6b5fc8c-mznzj" Jul 13 22:38:28.578648 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:38:28.578628 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5f7bf\" (UniqueName: \"kubernetes.io/projected/4a0e4a31-336c-4ac7-8ad8-9b2ea689099c-kube-api-access-5f7bf\") pod \"e2e-derive-036ddcf7-openshift-default-86c6b5fc8c-mznzj\" (UID: \"4a0e4a31-336c-4ac7-8ad8-9b2ea689099c\") " pod="openshift-ingress/e2e-derive-036ddcf7-openshift-default-86c6b5fc8c-mznzj" Jul 13 22:38:28.657394 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:38:28.657353 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-036ddcf7-openshift-default-86c6b5fc8c-mznzj" Jul 13 22:38:28.790437 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:38:28.790412 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-derive-036ddcf7-openshift-default-86c6b5fc8c-mznzj"] Jul 13 22:38:28.792805 ip-10-0-133-179 kubenswrapper[2583]: W0713 22:38:28.792767 2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4a0e4a31_336c_4ac7_8ad8_9b2ea689099c.slice/crio-6a03b31598a9026fd769db07d7977ac6abdd08e8cf1f5dc1510282f3aa26075a WatchSource:0}: Error finding container 6a03b31598a9026fd769db07d7977ac6abdd08e8cf1f5dc1510282f3aa26075a: Status 404 returned error can't find the container with id 6a03b31598a9026fd769db07d7977ac6abdd08e8cf1f5dc1510282f3aa26075a Jul 13 22:38:28.795603 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:38:28.795533 2583 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 13 22:38:28.795695 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:38:28.795634 2583 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 13 22:38:28.795695 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:38:28.795677 2583 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 13 22:38:29.297776 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:38:29.297740 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-036ddcf7-openshift-default-86c6b5fc8c-mznzj" event={"ID":"4a0e4a31-336c-4ac7-8ad8-9b2ea689099c","Type":"ContainerStarted","Data":"4b4acac211baef15c55980461061210aa8812e6d4259ff68e41d93c470e55e17"} Jul 13 22:38:29.298174 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:38:29.297791 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-036ddcf7-openshift-default-86c6b5fc8c-mznzj" event={"ID":"4a0e4a31-336c-4ac7-8ad8-9b2ea689099c","Type":"ContainerStarted","Data":"6a03b31598a9026fd769db07d7977ac6abdd08e8cf1f5dc1510282f3aa26075a"} Jul 13 22:38:29.319275 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:38:29.319221 2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-derive-036ddcf7-openshift-default-86c6b5fc8c-mznzj" podStartSLOduration=1.319199873 podStartE2EDuration="1.319199873s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-13 22:38:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 22:38:29.317281153 +0000 UTC m=+1504.565196502" watchObservedRunningTime="2026-07-13 22:38:29.319199873 +0000 UTC m=+1504.567115222" Jul 13 22:38:29.389403 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:38:29.389366 2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="df056db4-2e39-44b9-8fa6-d50e1d7abf65" path="/var/lib/kubelet/pods/df056db4-2e39-44b9-8fa6-d50e1d7abf65/volumes" Jul 13 22:38:29.652681 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:38:29.652593 2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-036ddcf7-7f4986dcdf-9lsq9"] Jul 13 22:38:29.654921 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:38:29.654903 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-036ddcf7-7f4986dcdf-9lsq9" Jul 13 22:38:29.657518 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:38:29.657487 2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-derive-036ddcf7-openshift-default-86c6b5fc8c-mznzj" Jul 13 22:38:29.657790 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:38:29.657776 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-derive-036ddcf7-serving-cert\"" Jul 13 22:38:29.663295 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:38:29.662952 2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-derive-036ddcf7-openshift-default-86c6b5fc8c-mznzj" Jul 13 22:38:29.665141 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:38:29.665120 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-036ddcf7-7f4986dcdf-9lsq9"] Jul 13 22:38:29.781046 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:38:29.781001 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2c8da727-2ce4-41ce-af61-3a79ebdf6ec9-maas-api-tls\") pod \"maas-api-e2e-derive-036ddcf7-7f4986dcdf-9lsq9\" (UID: \"2c8da727-2ce4-41ce-af61-3a79ebdf6ec9\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-036ddcf7-7f4986dcdf-9lsq9" Jul 13 22:38:29.781241 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:38:29.781053 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2c8da727-2ce4-41ce-af61-3a79ebdf6ec9-openshift-service-ca\") pod \"maas-api-e2e-derive-036ddcf7-7f4986dcdf-9lsq9\" (UID: \"2c8da727-2ce4-41ce-af61-3a79ebdf6ec9\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-036ddcf7-7f4986dcdf-9lsq9" Jul 13 22:38:29.781334 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:38:29.781276 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fnd74\" (UniqueName: \"kubernetes.io/projected/2c8da727-2ce4-41ce-af61-3a79ebdf6ec9-kube-api-access-fnd74\") pod \"maas-api-e2e-derive-036ddcf7-7f4986dcdf-9lsq9\" (UID: \"2c8da727-2ce4-41ce-af61-3a79ebdf6ec9\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-036ddcf7-7f4986dcdf-9lsq9" Jul 13 22:38:29.881976 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:38:29.881936 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2c8da727-2ce4-41ce-af61-3a79ebdf6ec9-maas-api-tls\") pod \"maas-api-e2e-derive-036ddcf7-7f4986dcdf-9lsq9\" (UID: \"2c8da727-2ce4-41ce-af61-3a79ebdf6ec9\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-036ddcf7-7f4986dcdf-9lsq9" Jul 13 22:38:29.881976 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:38:29.881981 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2c8da727-2ce4-41ce-af61-3a79ebdf6ec9-openshift-service-ca\") pod \"maas-api-e2e-derive-036ddcf7-7f4986dcdf-9lsq9\" (UID: \"2c8da727-2ce4-41ce-af61-3a79ebdf6ec9\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-036ddcf7-7f4986dcdf-9lsq9" Jul 13 22:38:29.882205 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:38:29.882175 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fnd74\" (UniqueName: \"kubernetes.io/projected/2c8da727-2ce4-41ce-af61-3a79ebdf6ec9-kube-api-access-fnd74\") pod \"maas-api-e2e-derive-036ddcf7-7f4986dcdf-9lsq9\" (UID: \"2c8da727-2ce4-41ce-af61-3a79ebdf6ec9\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-036ddcf7-7f4986dcdf-9lsq9" Jul 13 22:38:29.882707 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:38:29.882688 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2c8da727-2ce4-41ce-af61-3a79ebdf6ec9-openshift-service-ca\") pod \"maas-api-e2e-derive-036ddcf7-7f4986dcdf-9lsq9\" (UID: \"2c8da727-2ce4-41ce-af61-3a79ebdf6ec9\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-036ddcf7-7f4986dcdf-9lsq9" Jul 13 22:38:29.884488 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:38:29.884466 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2c8da727-2ce4-41ce-af61-3a79ebdf6ec9-maas-api-tls\") pod \"maas-api-e2e-derive-036ddcf7-7f4986dcdf-9lsq9\" (UID: \"2c8da727-2ce4-41ce-af61-3a79ebdf6ec9\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-036ddcf7-7f4986dcdf-9lsq9" Jul 13 22:38:29.889238 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:38:29.889219 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fnd74\" (UniqueName: \"kubernetes.io/projected/2c8da727-2ce4-41ce-af61-3a79ebdf6ec9-kube-api-access-fnd74\") pod \"maas-api-e2e-derive-036ddcf7-7f4986dcdf-9lsq9\" (UID: \"2c8da727-2ce4-41ce-af61-3a79ebdf6ec9\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-036ddcf7-7f4986dcdf-9lsq9" Jul 13 22:38:29.968424 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:38:29.968326 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-036ddcf7-7f4986dcdf-9lsq9" Jul 13 22:38:30.104315 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:38:30.104289 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-036ddcf7-7f4986dcdf-9lsq9"] Jul 13 22:38:30.106849 ip-10-0-133-179 kubenswrapper[2583]: W0713 22:38:30.106822 2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2c8da727_2ce4_41ce_af61_3a79ebdf6ec9.slice/crio-8df77cd6958ac302172001daccd0405409cb03fe8d09965f9914e62b44e55d37 WatchSource:0}: Error finding container 8df77cd6958ac302172001daccd0405409cb03fe8d09965f9914e62b44e55d37: Status 404 returned error can't find the container with id 8df77cd6958ac302172001daccd0405409cb03fe8d09965f9914e62b44e55d37 Jul 13 22:38:30.302556 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:38:30.302520 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-036ddcf7-7f4986dcdf-9lsq9" event={"ID":"2c8da727-2ce4-41ce-af61-3a79ebdf6ec9","Type":"ContainerStarted","Data":"8df77cd6958ac302172001daccd0405409cb03fe8d09965f9914e62b44e55d37"} Jul 13 22:38:30.302959 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:38:30.302753 2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-derive-036ddcf7-openshift-default-86c6b5fc8c-mznzj" Jul 13 22:38:30.303760 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:38:30.303737 2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-derive-036ddcf7-openshift-default-86c6b5fc8c-mznzj" Jul 13 22:38:31.309087 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:38:31.309046 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-036ddcf7-7f4986dcdf-9lsq9" event={"ID":"2c8da727-2ce4-41ce-af61-3a79ebdf6ec9","Type":"ContainerStarted","Data":"7f121c18995814fabcb8521ebe71e1ab1e3a86acecf5346c83d212c20139212f"} Jul 13 22:38:31.309495 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:38:31.309144 2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-derive-036ddcf7-7f4986dcdf-9lsq9" Jul 13 22:38:31.327963 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:38:31.327885 2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-derive-036ddcf7-7f4986dcdf-9lsq9" podStartSLOduration=2.049588456 podStartE2EDuration="2.327866001s" totalImagesPullingTime="278.277545ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-13 22:38:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 22:38:31.326321251 +0000 UTC m=+1506.574236603" watchObservedRunningTime="2026-07-13 22:38:31.327866001 +0000 UTC m=+1506.575781369" Jul 13 22:38:34.090129 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:38:34.090093 2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-derive-036ddcf7-openshift-default-86c6b5fc8c-mznzj"] Jul 13 22:38:34.090550 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:38:34.090263 2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-derive-036ddcf7-openshift-default-86c6b5fc8c-mznzj" podUID="4a0e4a31-336c-4ac7-8ad8-9b2ea689099c" containerName="istio-proxy" containerID="cri-o://4b4acac211baef15c55980461061210aa8812e6d4259ff68e41d93c470e55e17" gracePeriod=30 Jul 13 22:38:34.320603 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:38:34.320551 2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-036ddcf7-7f4986dcdf-9lsq9"] Jul 13 22:38:34.320827 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:38:34.320798 2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-derive-036ddcf7-7f4986dcdf-9lsq9" podUID="2c8da727-2ce4-41ce-af61-3a79ebdf6ec9" containerName="maas-api" containerID="cri-o://7f121c18995814fabcb8521ebe71e1ab1e3a86acecf5346c83d212c20139212f" gracePeriod=30 Jul 13 22:38:34.600468 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:38:34.600441 2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-036ddcf7-7f4986dcdf-9lsq9" Jul 13 22:38:34.726122 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:38:34.726030 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2c8da727-2ce4-41ce-af61-3a79ebdf6ec9-maas-api-tls\") pod \"2c8da727-2ce4-41ce-af61-3a79ebdf6ec9\" (UID: \"2c8da727-2ce4-41ce-af61-3a79ebdf6ec9\") " Jul 13 22:38:34.726122 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:38:34.726122 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-fnd74\" (UniqueName: \"kubernetes.io/projected/2c8da727-2ce4-41ce-af61-3a79ebdf6ec9-kube-api-access-fnd74\") pod \"2c8da727-2ce4-41ce-af61-3a79ebdf6ec9\" (UID: \"2c8da727-2ce4-41ce-af61-3a79ebdf6ec9\") " Jul 13 22:38:34.726336 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:38:34.726165 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2c8da727-2ce4-41ce-af61-3a79ebdf6ec9-openshift-service-ca\") pod \"2c8da727-2ce4-41ce-af61-3a79ebdf6ec9\" (UID: \"2c8da727-2ce4-41ce-af61-3a79ebdf6ec9\") " Jul 13 22:38:34.726536 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:38:34.726510 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2c8da727-2ce4-41ce-af61-3a79ebdf6ec9-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "2c8da727-2ce4-41ce-af61-3a79ebdf6ec9" (UID: "2c8da727-2ce4-41ce-af61-3a79ebdf6ec9"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 13 22:38:34.728177 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:38:34.728149 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2c8da727-2ce4-41ce-af61-3a79ebdf6ec9-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "2c8da727-2ce4-41ce-af61-3a79ebdf6ec9" (UID: "2c8da727-2ce4-41ce-af61-3a79ebdf6ec9"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 13 22:38:34.728223 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:38:34.728174 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2c8da727-2ce4-41ce-af61-3a79ebdf6ec9-kube-api-access-fnd74" (OuterVolumeSpecName: "kube-api-access-fnd74") pod "2c8da727-2ce4-41ce-af61-3a79ebdf6ec9" (UID: "2c8da727-2ce4-41ce-af61-3a79ebdf6ec9"). InnerVolumeSpecName "kube-api-access-fnd74". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 13 22:38:34.830590 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:38:34.827500 2583 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2c8da727-2ce4-41ce-af61-3a79ebdf6ec9-maas-api-tls\") on node \"ip-10-0-133-179.ec2.internal\" DevicePath \"\"" Jul 13 22:38:34.830590 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:38:34.827547 2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fnd74\" (UniqueName: \"kubernetes.io/projected/2c8da727-2ce4-41ce-af61-3a79ebdf6ec9-kube-api-access-fnd74\") on node \"ip-10-0-133-179.ec2.internal\" DevicePath \"\"" Jul 13 22:38:34.830590 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:38:34.827600 2583 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2c8da727-2ce4-41ce-af61-3a79ebdf6ec9-openshift-service-ca\") on node \"ip-10-0-133-179.ec2.internal\" DevicePath \"\"" Jul 13 22:38:35.333124 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:38:35.333078 2583 generic.go:356] "Generic (PLEG): container finished" podID="2c8da727-2ce4-41ce-af61-3a79ebdf6ec9" containerID="7f121c18995814fabcb8521ebe71e1ab1e3a86acecf5346c83d212c20139212f" exitCode=0 Jul 13 22:38:35.333649 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:38:35.333170 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-036ddcf7-7f4986dcdf-9lsq9" event={"ID":"2c8da727-2ce4-41ce-af61-3a79ebdf6ec9","Type":"ContainerDied","Data":"7f121c18995814fabcb8521ebe71e1ab1e3a86acecf5346c83d212c20139212f"} Jul 13 22:38:35.333649 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:38:35.333257 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-036ddcf7-7f4986dcdf-9lsq9" event={"ID":"2c8da727-2ce4-41ce-af61-3a79ebdf6ec9","Type":"ContainerDied","Data":"8df77cd6958ac302172001daccd0405409cb03fe8d09965f9914e62b44e55d37"} Jul 13 22:38:35.333649 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:38:35.333289 2583 scope.go:117] "RemoveContainer" containerID="7f121c18995814fabcb8521ebe71e1ab1e3a86acecf5346c83d212c20139212f" Jul 13 22:38:35.333649 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:38:35.333463 2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-036ddcf7-7f4986dcdf-9lsq9" Jul 13 22:38:35.357716 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:38:35.357403 2583 scope.go:117] "RemoveContainer" containerID="7f121c18995814fabcb8521ebe71e1ab1e3a86acecf5346c83d212c20139212f" Jul 13 22:38:35.357918 ip-10-0-133-179 kubenswrapper[2583]: E0713 22:38:35.357835 2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7f121c18995814fabcb8521ebe71e1ab1e3a86acecf5346c83d212c20139212f\": container with ID starting with 7f121c18995814fabcb8521ebe71e1ab1e3a86acecf5346c83d212c20139212f not found: ID does not exist" containerID="7f121c18995814fabcb8521ebe71e1ab1e3a86acecf5346c83d212c20139212f" Jul 13 22:38:35.357918 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:38:35.357882 2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7f121c18995814fabcb8521ebe71e1ab1e3a86acecf5346c83d212c20139212f"} err="failed to get container status \"7f121c18995814fabcb8521ebe71e1ab1e3a86acecf5346c83d212c20139212f\": rpc error: code = NotFound desc = could not find container \"7f121c18995814fabcb8521ebe71e1ab1e3a86acecf5346c83d212c20139212f\": container with ID starting with 7f121c18995814fabcb8521ebe71e1ab1e3a86acecf5346c83d212c20139212f not found: ID does not exist" Jul 13 22:38:35.359945 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:38:35.359920 2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-036ddcf7-7f4986dcdf-9lsq9"] Jul 13 22:38:35.362748 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:38:35.362724 2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-036ddcf7-7f4986dcdf-9lsq9"] Jul 13 22:38:35.389483 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:38:35.389452 2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2c8da727-2ce4-41ce-af61-3a79ebdf6ec9" path="/var/lib/kubelet/pods/2c8da727-2ce4-41ce-af61-3a79ebdf6ec9/volumes" Jul 13 22:38:39.664920 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:38:39.664894 2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-036ddcf7-openshift-default-86c6b5fc8c-mznzj" Jul 13 22:38:39.778518 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:38:39.778484 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4a0e4a31-336c-4ac7-8ad8-9b2ea689099c-istio-envoy\") pod \"4a0e4a31-336c-4ac7-8ad8-9b2ea689099c\" (UID: \"4a0e4a31-336c-4ac7-8ad8-9b2ea689099c\") " Jul 13 22:38:39.778726 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:38:39.778551 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4a0e4a31-336c-4ac7-8ad8-9b2ea689099c-workload-socket\") pod \"4a0e4a31-336c-4ac7-8ad8-9b2ea689099c\" (UID: \"4a0e4a31-336c-4ac7-8ad8-9b2ea689099c\") " Jul 13 22:38:39.778726 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:38:39.778610 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4a0e4a31-336c-4ac7-8ad8-9b2ea689099c-istio-data\") pod \"4a0e4a31-336c-4ac7-8ad8-9b2ea689099c\" (UID: \"4a0e4a31-336c-4ac7-8ad8-9b2ea689099c\") " Jul 13 22:38:39.778726 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:38:39.778642 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4a0e4a31-336c-4ac7-8ad8-9b2ea689099c-istio-podinfo\") pod \"4a0e4a31-336c-4ac7-8ad8-9b2ea689099c\" (UID: \"4a0e4a31-336c-4ac7-8ad8-9b2ea689099c\") " Jul 13 22:38:39.778726 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:38:39.778682 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4a0e4a31-336c-4ac7-8ad8-9b2ea689099c-istiod-ca-cert\") pod \"4a0e4a31-336c-4ac7-8ad8-9b2ea689099c\" (UID: \"4a0e4a31-336c-4ac7-8ad8-9b2ea689099c\") " Jul 13 22:38:39.778726 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:38:39.778709 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4a0e4a31-336c-4ac7-8ad8-9b2ea689099c-workload-certs\") pod \"4a0e4a31-336c-4ac7-8ad8-9b2ea689099c\" (UID: \"4a0e4a31-336c-4ac7-8ad8-9b2ea689099c\") " Jul 13 22:38:39.778975 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:38:39.778736 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4a0e4a31-336c-4ac7-8ad8-9b2ea689099c-istio-token\") pod \"4a0e4a31-336c-4ac7-8ad8-9b2ea689099c\" (UID: \"4a0e4a31-336c-4ac7-8ad8-9b2ea689099c\") " Jul 13 22:38:39.778975 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:38:39.778785 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5f7bf\" (UniqueName: \"kubernetes.io/projected/4a0e4a31-336c-4ac7-8ad8-9b2ea689099c-kube-api-access-5f7bf\") pod \"4a0e4a31-336c-4ac7-8ad8-9b2ea689099c\" (UID: \"4a0e4a31-336c-4ac7-8ad8-9b2ea689099c\") " Jul 13 22:38:39.778975 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:38:39.778818 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4a0e4a31-336c-4ac7-8ad8-9b2ea689099c-credential-socket\") pod \"4a0e4a31-336c-4ac7-8ad8-9b2ea689099c\" (UID: \"4a0e4a31-336c-4ac7-8ad8-9b2ea689099c\") " Jul 13 22:38:39.778975 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:38:39.778861 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4a0e4a31-336c-4ac7-8ad8-9b2ea689099c-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "4a0e4a31-336c-4ac7-8ad8-9b2ea689099c" (UID: "4a0e4a31-336c-4ac7-8ad8-9b2ea689099c"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 13 22:38:39.779182 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:38:39.779027 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4a0e4a31-336c-4ac7-8ad8-9b2ea689099c-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "4a0e4a31-336c-4ac7-8ad8-9b2ea689099c" (UID: "4a0e4a31-336c-4ac7-8ad8-9b2ea689099c"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 13 22:38:39.779182 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:38:39.779125 2583 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4a0e4a31-336c-4ac7-8ad8-9b2ea689099c-workload-socket\") on node \"ip-10-0-133-179.ec2.internal\" DevicePath \"\"" Jul 13 22:38:39.779182 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:38:39.779143 2583 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4a0e4a31-336c-4ac7-8ad8-9b2ea689099c-workload-certs\") on node \"ip-10-0-133-179.ec2.internal\" DevicePath \"\"" Jul 13 22:38:39.779334 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:38:39.779288 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4a0e4a31-336c-4ac7-8ad8-9b2ea689099c-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "4a0e4a31-336c-4ac7-8ad8-9b2ea689099c" (UID: "4a0e4a31-336c-4ac7-8ad8-9b2ea689099c"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 13 22:38:39.779507 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:38:39.779488 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4a0e4a31-336c-4ac7-8ad8-9b2ea689099c-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "4a0e4a31-336c-4ac7-8ad8-9b2ea689099c" (UID: "4a0e4a31-336c-4ac7-8ad8-9b2ea689099c"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 13 22:38:39.779607 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:38:39.779511 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4a0e4a31-336c-4ac7-8ad8-9b2ea689099c-istio-data" (OuterVolumeSpecName: "istio-data") pod "4a0e4a31-336c-4ac7-8ad8-9b2ea689099c" (UID: "4a0e4a31-336c-4ac7-8ad8-9b2ea689099c"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 13 22:38:39.780900 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:38:39.780873 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4a0e4a31-336c-4ac7-8ad8-9b2ea689099c-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "4a0e4a31-336c-4ac7-8ad8-9b2ea689099c" (UID: "4a0e4a31-336c-4ac7-8ad8-9b2ea689099c"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 13 22:38:39.781026 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:38:39.781009 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4a0e4a31-336c-4ac7-8ad8-9b2ea689099c-kube-api-access-5f7bf" (OuterVolumeSpecName: "kube-api-access-5f7bf") pod "4a0e4a31-336c-4ac7-8ad8-9b2ea689099c" (UID: "4a0e4a31-336c-4ac7-8ad8-9b2ea689099c"). InnerVolumeSpecName "kube-api-access-5f7bf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 13 22:38:39.781330 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:38:39.781310 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4a0e4a31-336c-4ac7-8ad8-9b2ea689099c-istio-token" (OuterVolumeSpecName: "istio-token") pod "4a0e4a31-336c-4ac7-8ad8-9b2ea689099c" (UID: "4a0e4a31-336c-4ac7-8ad8-9b2ea689099c"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 13 22:38:39.781390 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:38:39.781303 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/4a0e4a31-336c-4ac7-8ad8-9b2ea689099c-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "4a0e4a31-336c-4ac7-8ad8-9b2ea689099c" (UID: "4a0e4a31-336c-4ac7-8ad8-9b2ea689099c"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 13 22:38:39.879974 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:38:39.879933 2583 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4a0e4a31-336c-4ac7-8ad8-9b2ea689099c-istio-data\") on node \"ip-10-0-133-179.ec2.internal\" DevicePath \"\"" Jul 13 22:38:39.879974 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:38:39.879964 2583 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4a0e4a31-336c-4ac7-8ad8-9b2ea689099c-istio-podinfo\") on node \"ip-10-0-133-179.ec2.internal\" DevicePath \"\"" Jul 13 22:38:39.879974 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:38:39.879974 2583 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4a0e4a31-336c-4ac7-8ad8-9b2ea689099c-istiod-ca-cert\") on node \"ip-10-0-133-179.ec2.internal\" DevicePath \"\"" Jul 13 22:38:39.879974 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:38:39.879982 2583 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4a0e4a31-336c-4ac7-8ad8-9b2ea689099c-istio-token\") on node \"ip-10-0-133-179.ec2.internal\" DevicePath \"\"" Jul 13 22:38:39.880246 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:38:39.879992 2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5f7bf\" (UniqueName: \"kubernetes.io/projected/4a0e4a31-336c-4ac7-8ad8-9b2ea689099c-kube-api-access-5f7bf\") on node \"ip-10-0-133-179.ec2.internal\" DevicePath \"\"" Jul 13 22:38:39.880246 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:38:39.880001 2583 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4a0e4a31-336c-4ac7-8ad8-9b2ea689099c-credential-socket\") on node \"ip-10-0-133-179.ec2.internal\" DevicePath \"\"" Jul 13 22:38:39.880246 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:38:39.880009 2583 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4a0e4a31-336c-4ac7-8ad8-9b2ea689099c-istio-envoy\") on node \"ip-10-0-133-179.ec2.internal\" DevicePath \"\"" Jul 13 22:38:40.357548 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:38:40.357511 2583 generic.go:356] "Generic (PLEG): container finished" podID="4a0e4a31-336c-4ac7-8ad8-9b2ea689099c" containerID="4b4acac211baef15c55980461061210aa8812e6d4259ff68e41d93c470e55e17" exitCode=0 Jul 13 22:38:40.357746 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:38:40.357554 2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-036ddcf7-openshift-default-86c6b5fc8c-mznzj" Jul 13 22:38:40.357746 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:38:40.357603 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-036ddcf7-openshift-default-86c6b5fc8c-mznzj" event={"ID":"4a0e4a31-336c-4ac7-8ad8-9b2ea689099c","Type":"ContainerDied","Data":"4b4acac211baef15c55980461061210aa8812e6d4259ff68e41d93c470e55e17"} Jul 13 22:38:40.357746 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:38:40.357659 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-036ddcf7-openshift-default-86c6b5fc8c-mznzj" event={"ID":"4a0e4a31-336c-4ac7-8ad8-9b2ea689099c","Type":"ContainerDied","Data":"6a03b31598a9026fd769db07d7977ac6abdd08e8cf1f5dc1510282f3aa26075a"} Jul 13 22:38:40.357746 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:38:40.357683 2583 scope.go:117] "RemoveContainer" containerID="4b4acac211baef15c55980461061210aa8812e6d4259ff68e41d93c470e55e17" Jul 13 22:38:40.366290 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:38:40.366273 2583 scope.go:117] "RemoveContainer" containerID="4b4acac211baef15c55980461061210aa8812e6d4259ff68e41d93c470e55e17" Jul 13 22:38:40.366583 ip-10-0-133-179 kubenswrapper[2583]: E0713 22:38:40.366549 2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4b4acac211baef15c55980461061210aa8812e6d4259ff68e41d93c470e55e17\": container with ID starting with 4b4acac211baef15c55980461061210aa8812e6d4259ff68e41d93c470e55e17 not found: ID does not exist" containerID="4b4acac211baef15c55980461061210aa8812e6d4259ff68e41d93c470e55e17" Jul 13 22:38:40.366631 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:38:40.366589 2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4b4acac211baef15c55980461061210aa8812e6d4259ff68e41d93c470e55e17"} err="failed to get container status \"4b4acac211baef15c55980461061210aa8812e6d4259ff68e41d93c470e55e17\": rpc error: code = NotFound desc = could not find container \"4b4acac211baef15c55980461061210aa8812e6d4259ff68e41d93c470e55e17\": container with ID starting with 4b4acac211baef15c55980461061210aa8812e6d4259ff68e41d93c470e55e17 not found: ID does not exist" Jul 13 22:38:40.378402 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:38:40.378366 2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-derive-036ddcf7-openshift-default-86c6b5fc8c-mznzj"] Jul 13 22:38:40.381757 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:38:40.381731 2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-derive-036ddcf7-openshift-default-86c6b5fc8c-mznzj"] Jul 13 22:38:41.389297 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:38:41.389254 2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4a0e4a31-336c-4ac7-8ad8-9b2ea689099c" path="/var/lib/kubelet/pods/4a0e4a31-336c-4ac7-8ad8-9b2ea689099c/volumes" Jul 13 22:40:15.326943 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:40:15.326901 2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-e2fd044e-openshift-default-b9d546d87-z9jcx"] Jul 13 22:40:15.327513 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:40:15.327357 2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2c8da727-2ce4-41ce-af61-3a79ebdf6ec9" containerName="maas-api" Jul 13 22:40:15.327513 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:40:15.327371 2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="2c8da727-2ce4-41ce-af61-3a79ebdf6ec9" containerName="maas-api" Jul 13 22:40:15.327513 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:40:15.327404 2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4a0e4a31-336c-4ac7-8ad8-9b2ea689099c" containerName="istio-proxy" Jul 13 22:40:15.327513 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:40:15.327413 2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="4a0e4a31-336c-4ac7-8ad8-9b2ea689099c" containerName="istio-proxy" Jul 13 22:40:15.327513 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:40:15.327481 2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="4a0e4a31-336c-4ac7-8ad8-9b2ea689099c" containerName="istio-proxy" Jul 13 22:40:15.327513 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:40:15.327491 2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="2c8da727-2ce4-41ce-af61-3a79ebdf6ec9" containerName="maas-api" Jul 13 22:40:15.330608 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:40:15.330583 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-e2fd044e-openshift-default-b9d546d87-z9jcx" Jul 13 22:40:15.332734 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:40:15.332709 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-mt-e2fd044e-openshift-default-dockercfg-lsc7m\"" Jul 13 22:40:15.353614 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:40:15.353563 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-756gf\" (UniqueName: \"kubernetes.io/projected/879462b7-2707-413e-a2f8-effcb8235b37-kube-api-access-756gf\") pod \"e2e-mt-e2fd044e-openshift-default-b9d546d87-z9jcx\" (UID: \"879462b7-2707-413e-a2f8-effcb8235b37\") " pod="openshift-ingress/e2e-mt-e2fd044e-openshift-default-b9d546d87-z9jcx" Jul 13 22:40:15.353787 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:40:15.353626 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/879462b7-2707-413e-a2f8-effcb8235b37-workload-certs\") pod \"e2e-mt-e2fd044e-openshift-default-b9d546d87-z9jcx\" (UID: \"879462b7-2707-413e-a2f8-effcb8235b37\") " pod="openshift-ingress/e2e-mt-e2fd044e-openshift-default-b9d546d87-z9jcx" Jul 13 22:40:15.353787 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:40:15.353649 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/879462b7-2707-413e-a2f8-effcb8235b37-credential-socket\") pod \"e2e-mt-e2fd044e-openshift-default-b9d546d87-z9jcx\" (UID: \"879462b7-2707-413e-a2f8-effcb8235b37\") " pod="openshift-ingress/e2e-mt-e2fd044e-openshift-default-b9d546d87-z9jcx" Jul 13 22:40:15.353787 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:40:15.353683 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/879462b7-2707-413e-a2f8-effcb8235b37-istio-data\") pod \"e2e-mt-e2fd044e-openshift-default-b9d546d87-z9jcx\" (UID: \"879462b7-2707-413e-a2f8-effcb8235b37\") " pod="openshift-ingress/e2e-mt-e2fd044e-openshift-default-b9d546d87-z9jcx" Jul 13 22:40:15.353787 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:40:15.353683 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-e2fd044e-openshift-default-b9d546d87-z9jcx"] Jul 13 22:40:15.353787 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:40:15.353768 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/879462b7-2707-413e-a2f8-effcb8235b37-istio-podinfo\") pod \"e2e-mt-e2fd044e-openshift-default-b9d546d87-z9jcx\" (UID: \"879462b7-2707-413e-a2f8-effcb8235b37\") " pod="openshift-ingress/e2e-mt-e2fd044e-openshift-default-b9d546d87-z9jcx" Jul 13 22:40:15.354040 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:40:15.353797 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/879462b7-2707-413e-a2f8-effcb8235b37-istio-token\") pod \"e2e-mt-e2fd044e-openshift-default-b9d546d87-z9jcx\" (UID: \"879462b7-2707-413e-a2f8-effcb8235b37\") " pod="openshift-ingress/e2e-mt-e2fd044e-openshift-default-b9d546d87-z9jcx" Jul 13 22:40:15.354040 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:40:15.353821 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/879462b7-2707-413e-a2f8-effcb8235b37-istiod-ca-cert\") pod \"e2e-mt-e2fd044e-openshift-default-b9d546d87-z9jcx\" (UID: \"879462b7-2707-413e-a2f8-effcb8235b37\") " pod="openshift-ingress/e2e-mt-e2fd044e-openshift-default-b9d546d87-z9jcx" Jul 13 22:40:15.354040 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:40:15.353852 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/879462b7-2707-413e-a2f8-effcb8235b37-istio-envoy\") pod \"e2e-mt-e2fd044e-openshift-default-b9d546d87-z9jcx\" (UID: \"879462b7-2707-413e-a2f8-effcb8235b37\") " pod="openshift-ingress/e2e-mt-e2fd044e-openshift-default-b9d546d87-z9jcx" Jul 13 22:40:15.354040 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:40:15.353911 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/879462b7-2707-413e-a2f8-effcb8235b37-workload-socket\") pod \"e2e-mt-e2fd044e-openshift-default-b9d546d87-z9jcx\" (UID: \"879462b7-2707-413e-a2f8-effcb8235b37\") " pod="openshift-ingress/e2e-mt-e2fd044e-openshift-default-b9d546d87-z9jcx" Jul 13 22:40:15.454784 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:40:15.454742 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/879462b7-2707-413e-a2f8-effcb8235b37-istio-podinfo\") pod \"e2e-mt-e2fd044e-openshift-default-b9d546d87-z9jcx\" (UID: \"879462b7-2707-413e-a2f8-effcb8235b37\") " pod="openshift-ingress/e2e-mt-e2fd044e-openshift-default-b9d546d87-z9jcx" Jul 13 22:40:15.454978 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:40:15.454789 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/879462b7-2707-413e-a2f8-effcb8235b37-istio-token\") pod \"e2e-mt-e2fd044e-openshift-default-b9d546d87-z9jcx\" (UID: \"879462b7-2707-413e-a2f8-effcb8235b37\") " pod="openshift-ingress/e2e-mt-e2fd044e-openshift-default-b9d546d87-z9jcx" Jul 13 22:40:15.454978 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:40:15.454819 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/879462b7-2707-413e-a2f8-effcb8235b37-istiod-ca-cert\") pod \"e2e-mt-e2fd044e-openshift-default-b9d546d87-z9jcx\" (UID: \"879462b7-2707-413e-a2f8-effcb8235b37\") " pod="openshift-ingress/e2e-mt-e2fd044e-openshift-default-b9d546d87-z9jcx" Jul 13 22:40:15.454978 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:40:15.454889 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/879462b7-2707-413e-a2f8-effcb8235b37-istio-envoy\") pod \"e2e-mt-e2fd044e-openshift-default-b9d546d87-z9jcx\" (UID: \"879462b7-2707-413e-a2f8-effcb8235b37\") " pod="openshift-ingress/e2e-mt-e2fd044e-openshift-default-b9d546d87-z9jcx" Jul 13 22:40:15.454978 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:40:15.454927 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/879462b7-2707-413e-a2f8-effcb8235b37-workload-socket\") pod \"e2e-mt-e2fd044e-openshift-default-b9d546d87-z9jcx\" (UID: \"879462b7-2707-413e-a2f8-effcb8235b37\") " pod="openshift-ingress/e2e-mt-e2fd044e-openshift-default-b9d546d87-z9jcx" Jul 13 22:40:15.455150 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:40:15.455036 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-756gf\" (UniqueName: \"kubernetes.io/projected/879462b7-2707-413e-a2f8-effcb8235b37-kube-api-access-756gf\") pod \"e2e-mt-e2fd044e-openshift-default-b9d546d87-z9jcx\" (UID: \"879462b7-2707-413e-a2f8-effcb8235b37\") " pod="openshift-ingress/e2e-mt-e2fd044e-openshift-default-b9d546d87-z9jcx" Jul 13 22:40:15.455150 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:40:15.455065 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/879462b7-2707-413e-a2f8-effcb8235b37-workload-certs\") pod \"e2e-mt-e2fd044e-openshift-default-b9d546d87-z9jcx\" (UID: \"879462b7-2707-413e-a2f8-effcb8235b37\") " pod="openshift-ingress/e2e-mt-e2fd044e-openshift-default-b9d546d87-z9jcx" Jul 13 22:40:15.455150 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:40:15.455092 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/879462b7-2707-413e-a2f8-effcb8235b37-credential-socket\") pod \"e2e-mt-e2fd044e-openshift-default-b9d546d87-z9jcx\" (UID: \"879462b7-2707-413e-a2f8-effcb8235b37\") " pod="openshift-ingress/e2e-mt-e2fd044e-openshift-default-b9d546d87-z9jcx" Jul 13 22:40:15.455150 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:40:15.455132 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/879462b7-2707-413e-a2f8-effcb8235b37-istio-data\") pod \"e2e-mt-e2fd044e-openshift-default-b9d546d87-z9jcx\" (UID: \"879462b7-2707-413e-a2f8-effcb8235b37\") " pod="openshift-ingress/e2e-mt-e2fd044e-openshift-default-b9d546d87-z9jcx" Jul 13 22:40:15.455709 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:40:15.455682 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/879462b7-2707-413e-a2f8-effcb8235b37-credential-socket\") pod \"e2e-mt-e2fd044e-openshift-default-b9d546d87-z9jcx\" (UID: \"879462b7-2707-413e-a2f8-effcb8235b37\") " pod="openshift-ingress/e2e-mt-e2fd044e-openshift-default-b9d546d87-z9jcx" Jul 13 22:40:15.455821 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:40:15.455726 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/879462b7-2707-413e-a2f8-effcb8235b37-workload-certs\") pod \"e2e-mt-e2fd044e-openshift-default-b9d546d87-z9jcx\" (UID: \"879462b7-2707-413e-a2f8-effcb8235b37\") " pod="openshift-ingress/e2e-mt-e2fd044e-openshift-default-b9d546d87-z9jcx" Jul 13 22:40:15.455881 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:40:15.455761 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/879462b7-2707-413e-a2f8-effcb8235b37-istio-data\") pod \"e2e-mt-e2fd044e-openshift-default-b9d546d87-z9jcx\" (UID: \"879462b7-2707-413e-a2f8-effcb8235b37\") " pod="openshift-ingress/e2e-mt-e2fd044e-openshift-default-b9d546d87-z9jcx" Jul 13 22:40:15.455976 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:40:15.455920 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/879462b7-2707-413e-a2f8-effcb8235b37-workload-socket\") pod \"e2e-mt-e2fd044e-openshift-default-b9d546d87-z9jcx\" (UID: \"879462b7-2707-413e-a2f8-effcb8235b37\") " pod="openshift-ingress/e2e-mt-e2fd044e-openshift-default-b9d546d87-z9jcx" Jul 13 22:40:15.455976 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:40:15.455961 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/879462b7-2707-413e-a2f8-effcb8235b37-istiod-ca-cert\") pod \"e2e-mt-e2fd044e-openshift-default-b9d546d87-z9jcx\" (UID: \"879462b7-2707-413e-a2f8-effcb8235b37\") " pod="openshift-ingress/e2e-mt-e2fd044e-openshift-default-b9d546d87-z9jcx" Jul 13 22:40:15.457552 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:40:15.457520 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/879462b7-2707-413e-a2f8-effcb8235b37-istio-envoy\") pod \"e2e-mt-e2fd044e-openshift-default-b9d546d87-z9jcx\" (UID: \"879462b7-2707-413e-a2f8-effcb8235b37\") " pod="openshift-ingress/e2e-mt-e2fd044e-openshift-default-b9d546d87-z9jcx" Jul 13 22:40:15.457552 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:40:15.457548 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/879462b7-2707-413e-a2f8-effcb8235b37-istio-podinfo\") pod \"e2e-mt-e2fd044e-openshift-default-b9d546d87-z9jcx\" (UID: \"879462b7-2707-413e-a2f8-effcb8235b37\") " pod="openshift-ingress/e2e-mt-e2fd044e-openshift-default-b9d546d87-z9jcx" Jul 13 22:40:15.462342 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:40:15.462323 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/879462b7-2707-413e-a2f8-effcb8235b37-istio-token\") pod \"e2e-mt-e2fd044e-openshift-default-b9d546d87-z9jcx\" (UID: \"879462b7-2707-413e-a2f8-effcb8235b37\") " pod="openshift-ingress/e2e-mt-e2fd044e-openshift-default-b9d546d87-z9jcx" Jul 13 22:40:15.462765 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:40:15.462744 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-756gf\" (UniqueName: \"kubernetes.io/projected/879462b7-2707-413e-a2f8-effcb8235b37-kube-api-access-756gf\") pod \"e2e-mt-e2fd044e-openshift-default-b9d546d87-z9jcx\" (UID: \"879462b7-2707-413e-a2f8-effcb8235b37\") " pod="openshift-ingress/e2e-mt-e2fd044e-openshift-default-b9d546d87-z9jcx" Jul 13 22:40:15.642563 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:40:15.642462 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-e2fd044e-openshift-default-b9d546d87-z9jcx" Jul 13 22:40:15.793866 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:40:15.793837 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-e2fd044e-openshift-default-b9d546d87-z9jcx"] Jul 13 22:40:15.798655 ip-10-0-133-179 kubenswrapper[2583]: W0713 22:40:15.798622 2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod879462b7_2707_413e_a2f8_effcb8235b37.slice/crio-0190ec8d544e6705d2bf5f1b21450842ea78a1eb4f55e249ae32f8c4dec35a9e WatchSource:0}: Error finding container 0190ec8d544e6705d2bf5f1b21450842ea78a1eb4f55e249ae32f8c4dec35a9e: Status 404 returned error can't find the container with id 0190ec8d544e6705d2bf5f1b21450842ea78a1eb4f55e249ae32f8c4dec35a9e Jul 13 22:40:15.800250 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:40:15.800214 2583 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 13 22:40:15.800317 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:40:15.800283 2583 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 13 22:40:15.800317 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:40:15.800312 2583 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 13 22:40:16.755776 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:40:16.755683 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-e2fd044e-openshift-default-b9d546d87-z9jcx" event={"ID":"879462b7-2707-413e-a2f8-effcb8235b37","Type":"ContainerStarted","Data":"1eb32241440ee541d999d6c187d42e84bed3b639f613757574c2838899f1ef4a"} Jul 13 22:40:16.755776 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:40:16.755737 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-e2fd044e-openshift-default-b9d546d87-z9jcx" event={"ID":"879462b7-2707-413e-a2f8-effcb8235b37","Type":"ContainerStarted","Data":"0190ec8d544e6705d2bf5f1b21450842ea78a1eb4f55e249ae32f8c4dec35a9e"} Jul 13 22:40:16.775925 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:40:16.775871 2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-e2fd044e-openshift-default-b9d546d87-z9jcx" podStartSLOduration=1.775848592 podStartE2EDuration="1.775848592s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-13 22:40:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 22:40:16.775025024 +0000 UTC m=+1612.022940387" watchObservedRunningTime="2026-07-13 22:40:16.775848592 +0000 UTC m=+1612.023763944" Jul 13 22:40:17.643349 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:40:17.643310 2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-e2fd044e-openshift-default-b9d546d87-z9jcx" Jul 13 22:40:17.647991 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:40:17.647966 2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-e2fd044e-openshift-default-b9d546d87-z9jcx" Jul 13 22:40:17.760124 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:40:17.760096 2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-e2fd044e-openshift-default-b9d546d87-z9jcx" Jul 13 22:40:17.761010 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:40:17.760989 2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-e2fd044e-openshift-default-b9d546d87-z9jcx" Jul 13 22:40:18.790980 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:40:18.790950 2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-e2fd044e-d9f44876b-z4kn8"] Jul 13 22:40:18.794203 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:40:18.794182 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-e2fd044e-d9f44876b-z4kn8" Jul 13 22:40:18.796349 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:40:18.796330 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-mt-e2fd044e-serving-cert\"" Jul 13 22:40:18.805897 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:40:18.805868 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-e2fd044e-d9f44876b-z4kn8"] Jul 13 22:40:18.884956 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:40:18.884917 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7jpsv\" (UniqueName: \"kubernetes.io/projected/0c2da8b9-8523-4e60-9b73-a3ecab69e63d-kube-api-access-7jpsv\") pod \"maas-api-e2e-mt-e2fd044e-d9f44876b-z4kn8\" (UID: \"0c2da8b9-8523-4e60-9b73-a3ecab69e63d\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-e2fd044e-d9f44876b-z4kn8" Jul 13 22:40:18.885427 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:40:18.885402 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0c2da8b9-8523-4e60-9b73-a3ecab69e63d-openshift-service-ca\") pod \"maas-api-e2e-mt-e2fd044e-d9f44876b-z4kn8\" (UID: \"0c2da8b9-8523-4e60-9b73-a3ecab69e63d\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-e2fd044e-d9f44876b-z4kn8" Jul 13 22:40:18.885650 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:40:18.885632 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0c2da8b9-8523-4e60-9b73-a3ecab69e63d-maas-api-tls\") pod \"maas-api-e2e-mt-e2fd044e-d9f44876b-z4kn8\" (UID: \"0c2da8b9-8523-4e60-9b73-a3ecab69e63d\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-e2fd044e-d9f44876b-z4kn8" Jul 13 22:40:18.986463 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:40:18.986419 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7jpsv\" (UniqueName: \"kubernetes.io/projected/0c2da8b9-8523-4e60-9b73-a3ecab69e63d-kube-api-access-7jpsv\") pod \"maas-api-e2e-mt-e2fd044e-d9f44876b-z4kn8\" (UID: \"0c2da8b9-8523-4e60-9b73-a3ecab69e63d\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-e2fd044e-d9f44876b-z4kn8" Jul 13 22:40:18.986463 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:40:18.986467 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0c2da8b9-8523-4e60-9b73-a3ecab69e63d-openshift-service-ca\") pod \"maas-api-e2e-mt-e2fd044e-d9f44876b-z4kn8\" (UID: \"0c2da8b9-8523-4e60-9b73-a3ecab69e63d\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-e2fd044e-d9f44876b-z4kn8" Jul 13 22:40:18.986774 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:40:18.986493 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0c2da8b9-8523-4e60-9b73-a3ecab69e63d-maas-api-tls\") pod \"maas-api-e2e-mt-e2fd044e-d9f44876b-z4kn8\" (UID: \"0c2da8b9-8523-4e60-9b73-a3ecab69e63d\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-e2fd044e-d9f44876b-z4kn8" Jul 13 22:40:18.987296 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:40:18.987276 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0c2da8b9-8523-4e60-9b73-a3ecab69e63d-openshift-service-ca\") pod \"maas-api-e2e-mt-e2fd044e-d9f44876b-z4kn8\" (UID: \"0c2da8b9-8523-4e60-9b73-a3ecab69e63d\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-e2fd044e-d9f44876b-z4kn8" Jul 13 22:40:18.988965 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:40:18.988942 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0c2da8b9-8523-4e60-9b73-a3ecab69e63d-maas-api-tls\") pod \"maas-api-e2e-mt-e2fd044e-d9f44876b-z4kn8\" (UID: \"0c2da8b9-8523-4e60-9b73-a3ecab69e63d\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-e2fd044e-d9f44876b-z4kn8" Jul 13 22:40:18.993989 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:40:18.993963 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7jpsv\" (UniqueName: \"kubernetes.io/projected/0c2da8b9-8523-4e60-9b73-a3ecab69e63d-kube-api-access-7jpsv\") pod \"maas-api-e2e-mt-e2fd044e-d9f44876b-z4kn8\" (UID: \"0c2da8b9-8523-4e60-9b73-a3ecab69e63d\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-e2fd044e-d9f44876b-z4kn8" Jul 13 22:40:19.105366 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:40:19.105267 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-e2fd044e-d9f44876b-z4kn8" Jul 13 22:40:19.240906 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:40:19.240863 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-e2fd044e-d9f44876b-z4kn8"] Jul 13 22:40:19.243293 ip-10-0-133-179 kubenswrapper[2583]: W0713 22:40:19.243267 2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0c2da8b9_8523_4e60_9b73_a3ecab69e63d.slice/crio-fb23678a0519add382a47d522a9172c949c41cae44bf438706e237f4014ad0b9 WatchSource:0}: Error finding container fb23678a0519add382a47d522a9172c949c41cae44bf438706e237f4014ad0b9: Status 404 returned error can't find the container with id fb23678a0519add382a47d522a9172c949c41cae44bf438706e237f4014ad0b9 Jul 13 22:40:19.769987 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:40:19.769949 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-e2fd044e-d9f44876b-z4kn8" event={"ID":"0c2da8b9-8523-4e60-9b73-a3ecab69e63d","Type":"ContainerStarted","Data":"f8f6f73f6788dd6ab05e7e59c688cb2e1b0bc2759b4f99ee70f2fd6bbab7f2ab"} Jul 13 22:40:19.770165 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:40:19.770001 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-e2fd044e-d9f44876b-z4kn8" event={"ID":"0c2da8b9-8523-4e60-9b73-a3ecab69e63d","Type":"ContainerStarted","Data":"fb23678a0519add382a47d522a9172c949c41cae44bf438706e237f4014ad0b9"} Jul 13 22:40:19.770165 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:40:19.770115 2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-e2fd044e-d9f44876b-z4kn8" Jul 13 22:40:19.789601 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:40:19.786646 2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-mt-e2fd044e-d9f44876b-z4kn8" podStartSLOduration=1.431531552 podStartE2EDuration="1.786621674s" totalImagesPullingTime="355.090122ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-13 22:40:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 22:40:19.784308707 +0000 UTC m=+1615.032224057" watchObservedRunningTime="2026-07-13 22:40:19.786621674 +0000 UTC m=+1615.034537024" Jul 13 22:40:25.785194 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:40:25.785159 2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-e2fd044e-d9f44876b-z4kn8" Jul 13 22:40:30.752085 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:40:30.752049 2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-e2fd044e-d9f44876b-z4kn8"] Jul 13 22:40:30.752583 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:40:30.752271 2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-mt-e2fd044e-d9f44876b-z4kn8" podUID="0c2da8b9-8523-4e60-9b73-a3ecab69e63d" containerName="maas-api" containerID="cri-o://f8f6f73f6788dd6ab05e7e59c688cb2e1b0bc2759b4f99ee70f2fd6bbab7f2ab" gracePeriod=30 Jul 13 22:40:30.779195 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:40:30.779139 2583 prober.go:120] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-e2e-mt-e2fd044e-d9f44876b-z4kn8" podUID="0c2da8b9-8523-4e60-9b73-a3ecab69e63d" containerName="maas-api" probeResult="failure" output="Get \"https://10.133.0.30:8443/health\": dial tcp 10.133.0.30:8443: connect: connection refused" Jul 13 22:40:31.002729 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:40:31.002661 2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-e2fd044e-d9f44876b-z4kn8" Jul 13 22:40:31.107834 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:40:31.107786 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7jpsv\" (UniqueName: \"kubernetes.io/projected/0c2da8b9-8523-4e60-9b73-a3ecab69e63d-kube-api-access-7jpsv\") pod \"0c2da8b9-8523-4e60-9b73-a3ecab69e63d\" (UID: \"0c2da8b9-8523-4e60-9b73-a3ecab69e63d\") " Jul 13 22:40:31.108040 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:40:31.107856 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0c2da8b9-8523-4e60-9b73-a3ecab69e63d-openshift-service-ca\") pod \"0c2da8b9-8523-4e60-9b73-a3ecab69e63d\" (UID: \"0c2da8b9-8523-4e60-9b73-a3ecab69e63d\") " Jul 13 22:40:31.108040 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:40:31.107892 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0c2da8b9-8523-4e60-9b73-a3ecab69e63d-maas-api-tls\") pod \"0c2da8b9-8523-4e60-9b73-a3ecab69e63d\" (UID: \"0c2da8b9-8523-4e60-9b73-a3ecab69e63d\") " Jul 13 22:40:31.108254 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:40:31.108222 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0c2da8b9-8523-4e60-9b73-a3ecab69e63d-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "0c2da8b9-8523-4e60-9b73-a3ecab69e63d" (UID: "0c2da8b9-8523-4e60-9b73-a3ecab69e63d"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 13 22:40:31.110042 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:40:31.110003 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0c2da8b9-8523-4e60-9b73-a3ecab69e63d-kube-api-access-7jpsv" (OuterVolumeSpecName: "kube-api-access-7jpsv") pod "0c2da8b9-8523-4e60-9b73-a3ecab69e63d" (UID: "0c2da8b9-8523-4e60-9b73-a3ecab69e63d"). InnerVolumeSpecName "kube-api-access-7jpsv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 13 22:40:31.110144 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:40:31.110070 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0c2da8b9-8523-4e60-9b73-a3ecab69e63d-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "0c2da8b9-8523-4e60-9b73-a3ecab69e63d" (UID: "0c2da8b9-8523-4e60-9b73-a3ecab69e63d"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 13 22:40:31.209484 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:40:31.209450 2583 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0c2da8b9-8523-4e60-9b73-a3ecab69e63d-openshift-service-ca\") on node \"ip-10-0-133-179.ec2.internal\" DevicePath \"\"" Jul 13 22:40:31.209484 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:40:31.209481 2583 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0c2da8b9-8523-4e60-9b73-a3ecab69e63d-maas-api-tls\") on node \"ip-10-0-133-179.ec2.internal\" DevicePath \"\"" Jul 13 22:40:31.209484 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:40:31.209493 2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7jpsv\" (UniqueName: \"kubernetes.io/projected/0c2da8b9-8523-4e60-9b73-a3ecab69e63d-kube-api-access-7jpsv\") on node \"ip-10-0-133-179.ec2.internal\" DevicePath \"\"" Jul 13 22:40:31.823686 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:40:31.823646 2583 generic.go:356] "Generic (PLEG): container finished" podID="0c2da8b9-8523-4e60-9b73-a3ecab69e63d" containerID="f8f6f73f6788dd6ab05e7e59c688cb2e1b0bc2759b4f99ee70f2fd6bbab7f2ab" exitCode=0 Jul 13 22:40:31.824156 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:40:31.823699 2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-e2fd044e-d9f44876b-z4kn8" Jul 13 22:40:31.824156 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:40:31.823722 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-e2fd044e-d9f44876b-z4kn8" event={"ID":"0c2da8b9-8523-4e60-9b73-a3ecab69e63d","Type":"ContainerDied","Data":"f8f6f73f6788dd6ab05e7e59c688cb2e1b0bc2759b4f99ee70f2fd6bbab7f2ab"} Jul 13 22:40:31.824156 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:40:31.823775 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-e2fd044e-d9f44876b-z4kn8" event={"ID":"0c2da8b9-8523-4e60-9b73-a3ecab69e63d","Type":"ContainerDied","Data":"fb23678a0519add382a47d522a9172c949c41cae44bf438706e237f4014ad0b9"} Jul 13 22:40:31.824156 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:40:31.823806 2583 scope.go:117] "RemoveContainer" containerID="f8f6f73f6788dd6ab05e7e59c688cb2e1b0bc2759b4f99ee70f2fd6bbab7f2ab" Jul 13 22:40:31.825437 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:40:31.825414 2583 generic.go:356] "Generic (PLEG): container finished" podID="879462b7-2707-413e-a2f8-effcb8235b37" containerID="1eb32241440ee541d999d6c187d42e84bed3b639f613757574c2838899f1ef4a" exitCode=0 Jul 13 22:40:31.825519 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:40:31.825455 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-e2fd044e-openshift-default-b9d546d87-z9jcx" event={"ID":"879462b7-2707-413e-a2f8-effcb8235b37","Type":"ContainerDied","Data":"1eb32241440ee541d999d6c187d42e84bed3b639f613757574c2838899f1ef4a"} Jul 13 22:40:31.825859 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:40:31.825824 2583 scope.go:117] "RemoveContainer" containerID="1eb32241440ee541d999d6c187d42e84bed3b639f613757574c2838899f1ef4a" Jul 13 22:40:31.826952 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:40:31.826920 2583 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 13 22:40:31.827067 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:40:31.826995 2583 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 13 22:40:31.827067 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:40:31.827032 2583 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 13 22:40:31.835167 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:40:31.835148 2583 scope.go:117] "RemoveContainer" containerID="f8f6f73f6788dd6ab05e7e59c688cb2e1b0bc2759b4f99ee70f2fd6bbab7f2ab" Jul 13 22:40:31.835452 ip-10-0-133-179 kubenswrapper[2583]: E0713 22:40:31.835432 2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f8f6f73f6788dd6ab05e7e59c688cb2e1b0bc2759b4f99ee70f2fd6bbab7f2ab\": container with ID starting with f8f6f73f6788dd6ab05e7e59c688cb2e1b0bc2759b4f99ee70f2fd6bbab7f2ab not found: ID does not exist" containerID="f8f6f73f6788dd6ab05e7e59c688cb2e1b0bc2759b4f99ee70f2fd6bbab7f2ab" Jul 13 22:40:31.835523 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:40:31.835467 2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f8f6f73f6788dd6ab05e7e59c688cb2e1b0bc2759b4f99ee70f2fd6bbab7f2ab"} err="failed to get container status \"f8f6f73f6788dd6ab05e7e59c688cb2e1b0bc2759b4f99ee70f2fd6bbab7f2ab\": rpc error: code = NotFound desc = could not find container \"f8f6f73f6788dd6ab05e7e59c688cb2e1b0bc2759b4f99ee70f2fd6bbab7f2ab\": container with ID starting with f8f6f73f6788dd6ab05e7e59c688cb2e1b0bc2759b4f99ee70f2fd6bbab7f2ab not found: ID does not exist" Jul 13 22:40:31.839384 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:40:31.839363 2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-e2fd044e-d9f44876b-z4kn8"] Jul 13 22:40:31.842694 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:40:31.842676 2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-e2fd044e-d9f44876b-z4kn8"] Jul 13 22:40:32.760210 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:40:32.760167 2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-e2fd044e-openshift-default-b9d546d87-z9jcx" Jul 13 22:40:32.846029 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:40:32.845994 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-e2fd044e-openshift-default-b9d546d87-z9jcx" event={"ID":"879462b7-2707-413e-a2f8-effcb8235b37","Type":"ContainerStarted","Data":"c54867e197a63e9b33af4707a60970065d7fbacb61ddf751e62586c5d9adb0d2"} Jul 13 22:40:33.390210 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:40:33.390173 2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0c2da8b9-8523-4e60-9b73-a3ecab69e63d" path="/var/lib/kubelet/pods/0c2da8b9-8523-4e60-9b73-a3ecab69e63d/volumes" Jul 13 22:40:33.646653 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:40:33.645953 2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-e2fd044e-openshift-default-b9d546d87-z9jcx" Jul 13 22:40:33.650594 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:40:33.647167 2583 patch_prober.go:28] interesting pod/e2e-mt-e2fd044e-openshift-default-b9d546d87-z9jcx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.29:15021/healthz/ready\": dial tcp 10.133.0.29:15021: connect: connection refused" start-of-body= Jul 13 22:40:33.650594 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:40:33.647232 2583 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-e2fd044e-openshift-default-b9d546d87-z9jcx" podUID="879462b7-2707-413e-a2f8-effcb8235b37" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.29:15021/healthz/ready\": dial tcp 10.133.0.29:15021: connect: connection refused" Jul 13 22:40:34.643349 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:40:34.643285 2583 patch_prober.go:28] interesting pod/e2e-mt-e2fd044e-openshift-default-b9d546d87-z9jcx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.29:15021/healthz/ready\": dial tcp 10.133.0.29:15021: connect: connection refused" start-of-body= Jul 13 22:40:34.643979 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:40:34.643942 2583 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-e2fd044e-openshift-default-b9d546d87-z9jcx" podUID="879462b7-2707-413e-a2f8-effcb8235b37" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.29:15021/healthz/ready\": dial tcp 10.133.0.29:15021: connect: connection refused" Jul 13 22:40:35.643718 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:40:35.643673 2583 patch_prober.go:28] interesting pod/e2e-mt-e2fd044e-openshift-default-b9d546d87-z9jcx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.29:15021/healthz/ready\": dial tcp 10.133.0.29:15021: connect: connection refused" start-of-body= Jul 13 22:40:35.644236 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:40:35.643745 2583 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-e2fd044e-openshift-default-b9d546d87-z9jcx" podUID="879462b7-2707-413e-a2f8-effcb8235b37" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.29:15021/healthz/ready\": dial tcp 10.133.0.29:15021: connect: connection refused" Jul 13 22:40:36.643231 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:40:36.643184 2583 patch_prober.go:28] interesting pod/e2e-mt-e2fd044e-openshift-default-b9d546d87-z9jcx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.29:15021/healthz/ready\": dial tcp 10.133.0.29:15021: connect: connection refused" start-of-body= Jul 13 22:40:36.643438 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:40:36.643251 2583 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-e2fd044e-openshift-default-b9d546d87-z9jcx" podUID="879462b7-2707-413e-a2f8-effcb8235b37" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.29:15021/healthz/ready\": dial tcp 10.133.0.29:15021: connect: connection refused" Jul 13 22:40:37.198465 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:40:37.198433 2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-e2fd044e-openshift-default-b9d546d87-z9jcx"] Jul 13 22:40:37.198894 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:40:37.198670 2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-e2fd044e-openshift-default-b9d546d87-z9jcx" podUID="879462b7-2707-413e-a2f8-effcb8235b37" containerName="istio-proxy" containerID="cri-o://c54867e197a63e9b33af4707a60970065d7fbacb61ddf751e62586c5d9adb0d2" gracePeriod=30 Jul 13 22:40:42.449873 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:40:42.449846 2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-e2fd044e-openshift-default-b9d546d87-z9jcx" Jul 13 22:40:42.622194 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:40:42.622155 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/879462b7-2707-413e-a2f8-effcb8235b37-istio-envoy\") pod \"879462b7-2707-413e-a2f8-effcb8235b37\" (UID: \"879462b7-2707-413e-a2f8-effcb8235b37\") " Jul 13 22:40:42.622401 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:40:42.622234 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/879462b7-2707-413e-a2f8-effcb8235b37-istio-podinfo\") pod \"879462b7-2707-413e-a2f8-effcb8235b37\" (UID: \"879462b7-2707-413e-a2f8-effcb8235b37\") " Jul 13 22:40:42.622401 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:40:42.622271 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/879462b7-2707-413e-a2f8-effcb8235b37-credential-socket\") pod \"879462b7-2707-413e-a2f8-effcb8235b37\" (UID: \"879462b7-2707-413e-a2f8-effcb8235b37\") " Jul 13 22:40:42.622401 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:40:42.622293 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/879462b7-2707-413e-a2f8-effcb8235b37-istiod-ca-cert\") pod \"879462b7-2707-413e-a2f8-effcb8235b37\" (UID: \"879462b7-2707-413e-a2f8-effcb8235b37\") " Jul 13 22:40:42.622401 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:40:42.622320 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/879462b7-2707-413e-a2f8-effcb8235b37-istio-data\") pod \"879462b7-2707-413e-a2f8-effcb8235b37\" (UID: \"879462b7-2707-413e-a2f8-effcb8235b37\") " Jul 13 22:40:42.622401 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:40:42.622345 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/879462b7-2707-413e-a2f8-effcb8235b37-istio-token\") pod \"879462b7-2707-413e-a2f8-effcb8235b37\" (UID: \"879462b7-2707-413e-a2f8-effcb8235b37\") " Jul 13 22:40:42.622401 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:40:42.622374 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/879462b7-2707-413e-a2f8-effcb8235b37-workload-certs\") pod \"879462b7-2707-413e-a2f8-effcb8235b37\" (UID: \"879462b7-2707-413e-a2f8-effcb8235b37\") " Jul 13 22:40:42.622747 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:40:42.622427 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-756gf\" (UniqueName: \"kubernetes.io/projected/879462b7-2707-413e-a2f8-effcb8235b37-kube-api-access-756gf\") pod \"879462b7-2707-413e-a2f8-effcb8235b37\" (UID: \"879462b7-2707-413e-a2f8-effcb8235b37\") " Jul 13 22:40:42.622747 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:40:42.622451 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/879462b7-2707-413e-a2f8-effcb8235b37-workload-socket\") pod \"879462b7-2707-413e-a2f8-effcb8235b37\" (UID: \"879462b7-2707-413e-a2f8-effcb8235b37\") " Jul 13 22:40:42.622747 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:40:42.622601 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/879462b7-2707-413e-a2f8-effcb8235b37-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "879462b7-2707-413e-a2f8-effcb8235b37" (UID: "879462b7-2707-413e-a2f8-effcb8235b37"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 13 22:40:42.622747 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:40:42.622728 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/879462b7-2707-413e-a2f8-effcb8235b37-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "879462b7-2707-413e-a2f8-effcb8235b37" (UID: "879462b7-2707-413e-a2f8-effcb8235b37"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 13 22:40:42.622933 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:40:42.622772 2583 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/879462b7-2707-413e-a2f8-effcb8235b37-credential-socket\") on node \"ip-10-0-133-179.ec2.internal\" DevicePath \"\"" Jul 13 22:40:42.622933 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:40:42.622803 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/879462b7-2707-413e-a2f8-effcb8235b37-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "879462b7-2707-413e-a2f8-effcb8235b37" (UID: "879462b7-2707-413e-a2f8-effcb8235b37"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 13 22:40:42.623039 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:40:42.622934 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/879462b7-2707-413e-a2f8-effcb8235b37-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "879462b7-2707-413e-a2f8-effcb8235b37" (UID: "879462b7-2707-413e-a2f8-effcb8235b37"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 13 22:40:42.623398 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:40:42.623367 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/879462b7-2707-413e-a2f8-effcb8235b37-istio-data" (OuterVolumeSpecName: "istio-data") pod "879462b7-2707-413e-a2f8-effcb8235b37" (UID: "879462b7-2707-413e-a2f8-effcb8235b37"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 13 22:40:42.624610 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:40:42.624584 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/879462b7-2707-413e-a2f8-effcb8235b37-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "879462b7-2707-413e-a2f8-effcb8235b37" (UID: "879462b7-2707-413e-a2f8-effcb8235b37"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 13 22:40:42.624700 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:40:42.624587 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/879462b7-2707-413e-a2f8-effcb8235b37-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "879462b7-2707-413e-a2f8-effcb8235b37" (UID: "879462b7-2707-413e-a2f8-effcb8235b37"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 13 22:40:42.624700 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:40:42.624648 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/879462b7-2707-413e-a2f8-effcb8235b37-istio-token" (OuterVolumeSpecName: "istio-token") pod "879462b7-2707-413e-a2f8-effcb8235b37" (UID: "879462b7-2707-413e-a2f8-effcb8235b37"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 13 22:40:42.624778 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:40:42.624731 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/879462b7-2707-413e-a2f8-effcb8235b37-kube-api-access-756gf" (OuterVolumeSpecName: "kube-api-access-756gf") pod "879462b7-2707-413e-a2f8-effcb8235b37" (UID: "879462b7-2707-413e-a2f8-effcb8235b37"). InnerVolumeSpecName "kube-api-access-756gf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 13 22:40:42.723994 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:40:42.723956 2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-756gf\" (UniqueName: \"kubernetes.io/projected/879462b7-2707-413e-a2f8-effcb8235b37-kube-api-access-756gf\") on node \"ip-10-0-133-179.ec2.internal\" DevicePath \"\"" Jul 13 22:40:42.723994 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:40:42.723987 2583 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/879462b7-2707-413e-a2f8-effcb8235b37-workload-socket\") on node \"ip-10-0-133-179.ec2.internal\" DevicePath \"\"" Jul 13 22:40:42.723994 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:40:42.723997 2583 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/879462b7-2707-413e-a2f8-effcb8235b37-istio-envoy\") on node \"ip-10-0-133-179.ec2.internal\" DevicePath \"\"" Jul 13 22:40:42.723994 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:40:42.724007 2583 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/879462b7-2707-413e-a2f8-effcb8235b37-istio-podinfo\") on node \"ip-10-0-133-179.ec2.internal\" DevicePath \"\"" Jul 13 22:40:42.724243 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:40:42.724015 2583 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/879462b7-2707-413e-a2f8-effcb8235b37-istiod-ca-cert\") on node \"ip-10-0-133-179.ec2.internal\" DevicePath \"\"" Jul 13 22:40:42.724243 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:40:42.724024 2583 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/879462b7-2707-413e-a2f8-effcb8235b37-istio-data\") on node \"ip-10-0-133-179.ec2.internal\" DevicePath \"\"" Jul 13 22:40:42.724243 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:40:42.724031 2583 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/879462b7-2707-413e-a2f8-effcb8235b37-istio-token\") on node \"ip-10-0-133-179.ec2.internal\" DevicePath \"\"" Jul 13 22:40:42.724243 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:40:42.724038 2583 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/879462b7-2707-413e-a2f8-effcb8235b37-workload-certs\") on node \"ip-10-0-133-179.ec2.internal\" DevicePath \"\"" Jul 13 22:40:42.892769 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:40:42.892684 2583 generic.go:356] "Generic (PLEG): container finished" podID="879462b7-2707-413e-a2f8-effcb8235b37" containerID="c54867e197a63e9b33af4707a60970065d7fbacb61ddf751e62586c5d9adb0d2" exitCode=0 Jul 13 22:40:42.892769 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:40:42.892726 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-e2fd044e-openshift-default-b9d546d87-z9jcx" event={"ID":"879462b7-2707-413e-a2f8-effcb8235b37","Type":"ContainerDied","Data":"c54867e197a63e9b33af4707a60970065d7fbacb61ddf751e62586c5d9adb0d2"} Jul 13 22:40:42.892769 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:40:42.892733 2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-e2fd044e-openshift-default-b9d546d87-z9jcx" Jul 13 22:40:42.892769 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:40:42.892770 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-e2fd044e-openshift-default-b9d546d87-z9jcx" event={"ID":"879462b7-2707-413e-a2f8-effcb8235b37","Type":"ContainerDied","Data":"0190ec8d544e6705d2bf5f1b21450842ea78a1eb4f55e249ae32f8c4dec35a9e"} Jul 13 22:40:42.893007 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:40:42.892793 2583 scope.go:117] "RemoveContainer" containerID="c54867e197a63e9b33af4707a60970065d7fbacb61ddf751e62586c5d9adb0d2" Jul 13 22:40:42.902052 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:40:42.902026 2583 scope.go:117] "RemoveContainer" containerID="1eb32241440ee541d999d6c187d42e84bed3b639f613757574c2838899f1ef4a" Jul 13 22:40:42.910023 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:40:42.910005 2583 scope.go:117] "RemoveContainer" containerID="c54867e197a63e9b33af4707a60970065d7fbacb61ddf751e62586c5d9adb0d2" Jul 13 22:40:42.910309 ip-10-0-133-179 kubenswrapper[2583]: E0713 22:40:42.910286 2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c54867e197a63e9b33af4707a60970065d7fbacb61ddf751e62586c5d9adb0d2\": container with ID starting with c54867e197a63e9b33af4707a60970065d7fbacb61ddf751e62586c5d9adb0d2 not found: ID does not exist" containerID="c54867e197a63e9b33af4707a60970065d7fbacb61ddf751e62586c5d9adb0d2" Jul 13 22:40:42.910379 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:40:42.910323 2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c54867e197a63e9b33af4707a60970065d7fbacb61ddf751e62586c5d9adb0d2"} err="failed to get container status \"c54867e197a63e9b33af4707a60970065d7fbacb61ddf751e62586c5d9adb0d2\": rpc error: code = NotFound desc = could not find container \"c54867e197a63e9b33af4707a60970065d7fbacb61ddf751e62586c5d9adb0d2\": container with ID starting with c54867e197a63e9b33af4707a60970065d7fbacb61ddf751e62586c5d9adb0d2 not found: ID does not exist" Jul 13 22:40:42.910379 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:40:42.910347 2583 scope.go:117] "RemoveContainer" containerID="1eb32241440ee541d999d6c187d42e84bed3b639f613757574c2838899f1ef4a" Jul 13 22:40:42.910644 ip-10-0-133-179 kubenswrapper[2583]: E0713 22:40:42.910627 2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1eb32241440ee541d999d6c187d42e84bed3b639f613757574c2838899f1ef4a\": container with ID starting with 1eb32241440ee541d999d6c187d42e84bed3b639f613757574c2838899f1ef4a not found: ID does not exist" containerID="1eb32241440ee541d999d6c187d42e84bed3b639f613757574c2838899f1ef4a" Jul 13 22:40:42.910688 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:40:42.910654 2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1eb32241440ee541d999d6c187d42e84bed3b639f613757574c2838899f1ef4a"} err="failed to get container status \"1eb32241440ee541d999d6c187d42e84bed3b639f613757574c2838899f1ef4a\": rpc error: code = NotFound desc = could not find container \"1eb32241440ee541d999d6c187d42e84bed3b639f613757574c2838899f1ef4a\": container with ID starting with 1eb32241440ee541d999d6c187d42e84bed3b639f613757574c2838899f1ef4a not found: ID does not exist" Jul 13 22:40:42.921076 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:40:42.921043 2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-e2fd044e-openshift-default-b9d546d87-z9jcx"] Jul 13 22:40:42.922217 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:40:42.922198 2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-e2fd044e-openshift-default-b9d546d87-z9jcx"] Jul 13 22:40:43.389069 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:40:43.389033 2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="879462b7-2707-413e-a2f8-effcb8235b37" path="/var/lib/kubelet/pods/879462b7-2707-413e-a2f8-effcb8235b37/volumes" Jul 13 22:40:56.863184 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:40:56.863146 2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-6381cdd6-openshift-default-7487bfbcf5-vxwds"] Jul 13 22:40:56.863766 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:40:56.863446 2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="879462b7-2707-413e-a2f8-effcb8235b37" containerName="istio-proxy" Jul 13 22:40:56.863766 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:40:56.863457 2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="879462b7-2707-413e-a2f8-effcb8235b37" containerName="istio-proxy" Jul 13 22:40:56.863766 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:40:56.863466 2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="879462b7-2707-413e-a2f8-effcb8235b37" containerName="istio-proxy" Jul 13 22:40:56.863766 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:40:56.863472 2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="879462b7-2707-413e-a2f8-effcb8235b37" containerName="istio-proxy" Jul 13 22:40:56.863766 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:40:56.863483 2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0c2da8b9-8523-4e60-9b73-a3ecab69e63d" containerName="maas-api" Jul 13 22:40:56.863766 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:40:56.863490 2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="0c2da8b9-8523-4e60-9b73-a3ecab69e63d" containerName="maas-api" Jul 13 22:40:56.863766 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:40:56.863532 2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="879462b7-2707-413e-a2f8-effcb8235b37" containerName="istio-proxy" Jul 13 22:40:56.863766 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:40:56.863542 2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="879462b7-2707-413e-a2f8-effcb8235b37" containerName="istio-proxy" Jul 13 22:40:56.863766 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:40:56.863548 2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="0c2da8b9-8523-4e60-9b73-a3ecab69e63d" containerName="maas-api" Jul 13 22:40:56.865308 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:40:56.865292 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-6381cdd6-openshift-default-7487bfbcf5-vxwds" Jul 13 22:40:56.867535 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:40:56.867510 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-mt-6381cdd6-openshift-default-dockercfg-49mhq\"" Jul 13 22:40:56.876849 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:40:56.876826 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-6381cdd6-openshift-default-7487bfbcf5-vxwds"] Jul 13 22:40:56.940762 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:40:56.940725 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/618c7b21-2285-401a-b8a4-18dd32b98671-credential-socket\") pod \"e2e-mt-6381cdd6-openshift-default-7487bfbcf5-vxwds\" (UID: \"618c7b21-2285-401a-b8a4-18dd32b98671\") " pod="openshift-ingress/e2e-mt-6381cdd6-openshift-default-7487bfbcf5-vxwds" Jul 13 22:40:56.940762 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:40:56.940763 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/618c7b21-2285-401a-b8a4-18dd32b98671-istio-envoy\") pod \"e2e-mt-6381cdd6-openshift-default-7487bfbcf5-vxwds\" (UID: \"618c7b21-2285-401a-b8a4-18dd32b98671\") " pod="openshift-ingress/e2e-mt-6381cdd6-openshift-default-7487bfbcf5-vxwds" Jul 13 22:40:56.941004 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:40:56.940786 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/618c7b21-2285-401a-b8a4-18dd32b98671-istio-data\") pod \"e2e-mt-6381cdd6-openshift-default-7487bfbcf5-vxwds\" (UID: \"618c7b21-2285-401a-b8a4-18dd32b98671\") " pod="openshift-ingress/e2e-mt-6381cdd6-openshift-default-7487bfbcf5-vxwds" Jul 13 22:40:56.941004 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:40:56.940847 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/618c7b21-2285-401a-b8a4-18dd32b98671-istio-podinfo\") pod \"e2e-mt-6381cdd6-openshift-default-7487bfbcf5-vxwds\" (UID: \"618c7b21-2285-401a-b8a4-18dd32b98671\") " pod="openshift-ingress/e2e-mt-6381cdd6-openshift-default-7487bfbcf5-vxwds" Jul 13 22:40:56.941004 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:40:56.940878 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/618c7b21-2285-401a-b8a4-18dd32b98671-istiod-ca-cert\") pod \"e2e-mt-6381cdd6-openshift-default-7487bfbcf5-vxwds\" (UID: \"618c7b21-2285-401a-b8a4-18dd32b98671\") " pod="openshift-ingress/e2e-mt-6381cdd6-openshift-default-7487bfbcf5-vxwds" Jul 13 22:40:56.941004 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:40:56.940912 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/618c7b21-2285-401a-b8a4-18dd32b98671-workload-socket\") pod \"e2e-mt-6381cdd6-openshift-default-7487bfbcf5-vxwds\" (UID: \"618c7b21-2285-401a-b8a4-18dd32b98671\") " pod="openshift-ingress/e2e-mt-6381cdd6-openshift-default-7487bfbcf5-vxwds" Jul 13 22:40:56.941004 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:40:56.940941 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pvrw5\" (UniqueName: \"kubernetes.io/projected/618c7b21-2285-401a-b8a4-18dd32b98671-kube-api-access-pvrw5\") pod \"e2e-mt-6381cdd6-openshift-default-7487bfbcf5-vxwds\" (UID: \"618c7b21-2285-401a-b8a4-18dd32b98671\") " pod="openshift-ingress/e2e-mt-6381cdd6-openshift-default-7487bfbcf5-vxwds" Jul 13 22:40:56.941004 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:40:56.940980 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/618c7b21-2285-401a-b8a4-18dd32b98671-workload-certs\") pod \"e2e-mt-6381cdd6-openshift-default-7487bfbcf5-vxwds\" (UID: \"618c7b21-2285-401a-b8a4-18dd32b98671\") " pod="openshift-ingress/e2e-mt-6381cdd6-openshift-default-7487bfbcf5-vxwds" Jul 13 22:40:56.941218 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:40:56.941013 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/618c7b21-2285-401a-b8a4-18dd32b98671-istio-token\") pod \"e2e-mt-6381cdd6-openshift-default-7487bfbcf5-vxwds\" (UID: \"618c7b21-2285-401a-b8a4-18dd32b98671\") " pod="openshift-ingress/e2e-mt-6381cdd6-openshift-default-7487bfbcf5-vxwds" Jul 13 22:40:57.041903 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:40:57.041862 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pvrw5\" (UniqueName: \"kubernetes.io/projected/618c7b21-2285-401a-b8a4-18dd32b98671-kube-api-access-pvrw5\") pod \"e2e-mt-6381cdd6-openshift-default-7487bfbcf5-vxwds\" (UID: \"618c7b21-2285-401a-b8a4-18dd32b98671\") " pod="openshift-ingress/e2e-mt-6381cdd6-openshift-default-7487bfbcf5-vxwds" Jul 13 22:40:57.041903 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:40:57.041902 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/618c7b21-2285-401a-b8a4-18dd32b98671-workload-certs\") pod \"e2e-mt-6381cdd6-openshift-default-7487bfbcf5-vxwds\" (UID: \"618c7b21-2285-401a-b8a4-18dd32b98671\") " pod="openshift-ingress/e2e-mt-6381cdd6-openshift-default-7487bfbcf5-vxwds" Jul 13 22:40:57.042111 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:40:57.041926 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/618c7b21-2285-401a-b8a4-18dd32b98671-istio-token\") pod \"e2e-mt-6381cdd6-openshift-default-7487bfbcf5-vxwds\" (UID: \"618c7b21-2285-401a-b8a4-18dd32b98671\") " pod="openshift-ingress/e2e-mt-6381cdd6-openshift-default-7487bfbcf5-vxwds" Jul 13 22:40:57.042111 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:40:57.041957 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/618c7b21-2285-401a-b8a4-18dd32b98671-credential-socket\") pod \"e2e-mt-6381cdd6-openshift-default-7487bfbcf5-vxwds\" (UID: \"618c7b21-2285-401a-b8a4-18dd32b98671\") " pod="openshift-ingress/e2e-mt-6381cdd6-openshift-default-7487bfbcf5-vxwds" Jul 13 22:40:57.042225 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:40:57.042134 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/618c7b21-2285-401a-b8a4-18dd32b98671-istio-envoy\") pod \"e2e-mt-6381cdd6-openshift-default-7487bfbcf5-vxwds\" (UID: \"618c7b21-2285-401a-b8a4-18dd32b98671\") " pod="openshift-ingress/e2e-mt-6381cdd6-openshift-default-7487bfbcf5-vxwds" Jul 13 22:40:57.042225 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:40:57.042185 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/618c7b21-2285-401a-b8a4-18dd32b98671-istio-data\") pod \"e2e-mt-6381cdd6-openshift-default-7487bfbcf5-vxwds\" (UID: \"618c7b21-2285-401a-b8a4-18dd32b98671\") " pod="openshift-ingress/e2e-mt-6381cdd6-openshift-default-7487bfbcf5-vxwds" Jul 13 22:40:57.042314 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:40:57.042230 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/618c7b21-2285-401a-b8a4-18dd32b98671-istio-podinfo\") pod \"e2e-mt-6381cdd6-openshift-default-7487bfbcf5-vxwds\" (UID: \"618c7b21-2285-401a-b8a4-18dd32b98671\") " pod="openshift-ingress/e2e-mt-6381cdd6-openshift-default-7487bfbcf5-vxwds" Jul 13 22:40:57.042314 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:40:57.042258 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/618c7b21-2285-401a-b8a4-18dd32b98671-istiod-ca-cert\") pod \"e2e-mt-6381cdd6-openshift-default-7487bfbcf5-vxwds\" (UID: \"618c7b21-2285-401a-b8a4-18dd32b98671\") " pod="openshift-ingress/e2e-mt-6381cdd6-openshift-default-7487bfbcf5-vxwds" Jul 13 22:40:57.042314 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:40:57.042283 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/618c7b21-2285-401a-b8a4-18dd32b98671-workload-certs\") pod \"e2e-mt-6381cdd6-openshift-default-7487bfbcf5-vxwds\" (UID: \"618c7b21-2285-401a-b8a4-18dd32b98671\") " pod="openshift-ingress/e2e-mt-6381cdd6-openshift-default-7487bfbcf5-vxwds" Jul 13 22:40:57.042314 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:40:57.042310 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/618c7b21-2285-401a-b8a4-18dd32b98671-workload-socket\") pod \"e2e-mt-6381cdd6-openshift-default-7487bfbcf5-vxwds\" (UID: \"618c7b21-2285-401a-b8a4-18dd32b98671\") " pod="openshift-ingress/e2e-mt-6381cdd6-openshift-default-7487bfbcf5-vxwds" Jul 13 22:40:57.042521 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:40:57.042359 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/618c7b21-2285-401a-b8a4-18dd32b98671-credential-socket\") pod \"e2e-mt-6381cdd6-openshift-default-7487bfbcf5-vxwds\" (UID: \"618c7b21-2285-401a-b8a4-18dd32b98671\") " pod="openshift-ingress/e2e-mt-6381cdd6-openshift-default-7487bfbcf5-vxwds" Jul 13 22:40:57.042611 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:40:57.042593 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/618c7b21-2285-401a-b8a4-18dd32b98671-workload-socket\") pod \"e2e-mt-6381cdd6-openshift-default-7487bfbcf5-vxwds\" (UID: \"618c7b21-2285-401a-b8a4-18dd32b98671\") " pod="openshift-ingress/e2e-mt-6381cdd6-openshift-default-7487bfbcf5-vxwds" Jul 13 22:40:57.042666 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:40:57.042561 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/618c7b21-2285-401a-b8a4-18dd32b98671-istio-data\") pod \"e2e-mt-6381cdd6-openshift-default-7487bfbcf5-vxwds\" (UID: \"618c7b21-2285-401a-b8a4-18dd32b98671\") " pod="openshift-ingress/e2e-mt-6381cdd6-openshift-default-7487bfbcf5-vxwds" Jul 13 22:40:57.042919 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:40:57.042901 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/618c7b21-2285-401a-b8a4-18dd32b98671-istiod-ca-cert\") pod \"e2e-mt-6381cdd6-openshift-default-7487bfbcf5-vxwds\" (UID: \"618c7b21-2285-401a-b8a4-18dd32b98671\") " pod="openshift-ingress/e2e-mt-6381cdd6-openshift-default-7487bfbcf5-vxwds" Jul 13 22:40:57.044382 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:40:57.044352 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/618c7b21-2285-401a-b8a4-18dd32b98671-istio-envoy\") pod \"e2e-mt-6381cdd6-openshift-default-7487bfbcf5-vxwds\" (UID: \"618c7b21-2285-401a-b8a4-18dd32b98671\") " pod="openshift-ingress/e2e-mt-6381cdd6-openshift-default-7487bfbcf5-vxwds" Jul 13 22:40:57.044672 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:40:57.044653 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/618c7b21-2285-401a-b8a4-18dd32b98671-istio-podinfo\") pod \"e2e-mt-6381cdd6-openshift-default-7487bfbcf5-vxwds\" (UID: \"618c7b21-2285-401a-b8a4-18dd32b98671\") " pod="openshift-ingress/e2e-mt-6381cdd6-openshift-default-7487bfbcf5-vxwds" Jul 13 22:40:57.053154 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:40:57.053127 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/618c7b21-2285-401a-b8a4-18dd32b98671-istio-token\") pod \"e2e-mt-6381cdd6-openshift-default-7487bfbcf5-vxwds\" (UID: \"618c7b21-2285-401a-b8a4-18dd32b98671\") " pod="openshift-ingress/e2e-mt-6381cdd6-openshift-default-7487bfbcf5-vxwds" Jul 13 22:40:57.053314 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:40:57.053195 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pvrw5\" (UniqueName: \"kubernetes.io/projected/618c7b21-2285-401a-b8a4-18dd32b98671-kube-api-access-pvrw5\") pod \"e2e-mt-6381cdd6-openshift-default-7487bfbcf5-vxwds\" (UID: \"618c7b21-2285-401a-b8a4-18dd32b98671\") " pod="openshift-ingress/e2e-mt-6381cdd6-openshift-default-7487bfbcf5-vxwds" Jul 13 22:40:57.175941 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:40:57.175839 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-6381cdd6-openshift-default-7487bfbcf5-vxwds" Jul 13 22:40:57.314526 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:40:57.314490 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-6381cdd6-openshift-default-7487bfbcf5-vxwds"] Jul 13 22:40:57.317303 ip-10-0-133-179 kubenswrapper[2583]: W0713 22:40:57.317280 2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod618c7b21_2285_401a_b8a4_18dd32b98671.slice/crio-9cb248e0e838d9f8dfe197301b6064de26069893c2757e692cf52830e3bc6b2e WatchSource:0}: Error finding container 9cb248e0e838d9f8dfe197301b6064de26069893c2757e692cf52830e3bc6b2e: Status 404 returned error can't find the container with id 9cb248e0e838d9f8dfe197301b6064de26069893c2757e692cf52830e3bc6b2e Jul 13 22:40:57.319282 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:40:57.319246 2583 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 13 22:40:57.319353 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:40:57.319317 2583 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 13 22:40:57.319353 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:40:57.319344 2583 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 13 22:40:57.958528 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:40:57.958492 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-6381cdd6-openshift-default-7487bfbcf5-vxwds" event={"ID":"618c7b21-2285-401a-b8a4-18dd32b98671","Type":"ContainerStarted","Data":"aacdec7b0e136ef4d9016b5844f8ddd94809f7946a0110ab6b98c7b80ed88b16"} Jul 13 22:40:57.958952 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:40:57.958609 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-6381cdd6-openshift-default-7487bfbcf5-vxwds" event={"ID":"618c7b21-2285-401a-b8a4-18dd32b98671","Type":"ContainerStarted","Data":"9cb248e0e838d9f8dfe197301b6064de26069893c2757e692cf52830e3bc6b2e"} Jul 13 22:40:57.978188 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:40:57.978138 2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-6381cdd6-openshift-default-7487bfbcf5-vxwds" podStartSLOduration=1.978120925 podStartE2EDuration="1.978120925s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-13 22:40:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 22:40:57.97659453 +0000 UTC m=+1653.224509903" watchObservedRunningTime="2026-07-13 22:40:57.978120925 +0000 UTC m=+1653.226036351" Jul 13 22:40:58.176232 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:40:58.176190 2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-6381cdd6-openshift-default-7487bfbcf5-vxwds" Jul 13 22:40:58.181089 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:40:58.181062 2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-6381cdd6-openshift-default-7487bfbcf5-vxwds" Jul 13 22:40:58.963071 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:40:58.963038 2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-6381cdd6-openshift-default-7487bfbcf5-vxwds" Jul 13 22:40:58.964173 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:40:58.964151 2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-6381cdd6-openshift-default-7487bfbcf5-vxwds" Jul 13 22:40:59.948426 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:40:59.948393 2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-6381cdd6-6ddf876f9f-8bsrh"] Jul 13 22:40:59.951043 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:40:59.951024 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-6381cdd6-6ddf876f9f-8bsrh" Jul 13 22:40:59.953023 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:40:59.953002 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-mt-6381cdd6-serving-cert\"" Jul 13 22:40:59.961532 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:40:59.961485 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-6381cdd6-6ddf876f9f-8bsrh"] Jul 13 22:40:59.970256 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:40:59.970229 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1faef928-506a-43e0-97af-bb6ce8b9fda0-openshift-service-ca\") pod \"maas-api-e2e-mt-6381cdd6-6ddf876f9f-8bsrh\" (UID: \"1faef928-506a-43e0-97af-bb6ce8b9fda0\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-6381cdd6-6ddf876f9f-8bsrh" Jul 13 22:40:59.970624 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:40:59.970336 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1faef928-506a-43e0-97af-bb6ce8b9fda0-maas-api-tls\") pod \"maas-api-e2e-mt-6381cdd6-6ddf876f9f-8bsrh\" (UID: \"1faef928-506a-43e0-97af-bb6ce8b9fda0\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-6381cdd6-6ddf876f9f-8bsrh" Jul 13 22:40:59.970624 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:40:59.970414 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5w7jk\" (UniqueName: \"kubernetes.io/projected/1faef928-506a-43e0-97af-bb6ce8b9fda0-kube-api-access-5w7jk\") pod \"maas-api-e2e-mt-6381cdd6-6ddf876f9f-8bsrh\" (UID: \"1faef928-506a-43e0-97af-bb6ce8b9fda0\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-6381cdd6-6ddf876f9f-8bsrh" Jul 13 22:41:00.071358 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:41:00.071319 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1faef928-506a-43e0-97af-bb6ce8b9fda0-maas-api-tls\") pod \"maas-api-e2e-mt-6381cdd6-6ddf876f9f-8bsrh\" (UID: \"1faef928-506a-43e0-97af-bb6ce8b9fda0\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-6381cdd6-6ddf876f9f-8bsrh" Jul 13 22:41:00.071550 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:41:00.071400 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5w7jk\" (UniqueName: \"kubernetes.io/projected/1faef928-506a-43e0-97af-bb6ce8b9fda0-kube-api-access-5w7jk\") pod \"maas-api-e2e-mt-6381cdd6-6ddf876f9f-8bsrh\" (UID: \"1faef928-506a-43e0-97af-bb6ce8b9fda0\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-6381cdd6-6ddf876f9f-8bsrh" Jul 13 22:41:00.071550 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:41:00.071448 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1faef928-506a-43e0-97af-bb6ce8b9fda0-openshift-service-ca\") pod \"maas-api-e2e-mt-6381cdd6-6ddf876f9f-8bsrh\" (UID: \"1faef928-506a-43e0-97af-bb6ce8b9fda0\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-6381cdd6-6ddf876f9f-8bsrh" Jul 13 22:41:00.072188 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:41:00.072166 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1faef928-506a-43e0-97af-bb6ce8b9fda0-openshift-service-ca\") pod \"maas-api-e2e-mt-6381cdd6-6ddf876f9f-8bsrh\" (UID: \"1faef928-506a-43e0-97af-bb6ce8b9fda0\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-6381cdd6-6ddf876f9f-8bsrh" Jul 13 22:41:00.073835 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:41:00.073819 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1faef928-506a-43e0-97af-bb6ce8b9fda0-maas-api-tls\") pod \"maas-api-e2e-mt-6381cdd6-6ddf876f9f-8bsrh\" (UID: \"1faef928-506a-43e0-97af-bb6ce8b9fda0\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-6381cdd6-6ddf876f9f-8bsrh" Jul 13 22:41:00.080392 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:41:00.080364 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5w7jk\" (UniqueName: \"kubernetes.io/projected/1faef928-506a-43e0-97af-bb6ce8b9fda0-kube-api-access-5w7jk\") pod \"maas-api-e2e-mt-6381cdd6-6ddf876f9f-8bsrh\" (UID: \"1faef928-506a-43e0-97af-bb6ce8b9fda0\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-6381cdd6-6ddf876f9f-8bsrh" Jul 13 22:41:00.262843 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:41:00.262751 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-6381cdd6-6ddf876f9f-8bsrh" Jul 13 22:41:00.394814 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:41:00.394782 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-6381cdd6-6ddf876f9f-8bsrh"] Jul 13 22:41:00.397909 ip-10-0-133-179 kubenswrapper[2583]: W0713 22:41:00.397878 2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1faef928_506a_43e0_97af_bb6ce8b9fda0.slice/crio-8ded0249b86270851acc61c505b957a7521993597064142b1ae59a34051a544e WatchSource:0}: Error finding container 8ded0249b86270851acc61c505b957a7521993597064142b1ae59a34051a544e: Status 404 returned error can't find the container with id 8ded0249b86270851acc61c505b957a7521993597064142b1ae59a34051a544e Jul 13 22:41:00.973646 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:41:00.973606 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-6381cdd6-6ddf876f9f-8bsrh" event={"ID":"1faef928-506a-43e0-97af-bb6ce8b9fda0","Type":"ContainerStarted","Data":"ba6cf0cb9e044d3898c9d85a3942c6b79037b7da24182685552d892266e477ea"} Jul 13 22:41:00.974123 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:41:00.973665 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-6381cdd6-6ddf876f9f-8bsrh" event={"ID":"1faef928-506a-43e0-97af-bb6ce8b9fda0","Type":"ContainerStarted","Data":"8ded0249b86270851acc61c505b957a7521993597064142b1ae59a34051a544e"} Jul 13 22:41:00.998086 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:41:00.998035 2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-mt-6381cdd6-6ddf876f9f-8bsrh" podStartSLOduration=1.597747171 podStartE2EDuration="1.998019157s" totalImagesPullingTime="400.271986ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-13 22:40:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 22:41:00.996006644 +0000 UTC m=+1656.243921994" watchObservedRunningTime="2026-07-13 22:41:00.998019157 +0000 UTC m=+1656.245934506" Jul 13 22:41:01.978409 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:41:01.978366 2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-6381cdd6-6ddf876f9f-8bsrh" Jul 13 22:41:07.988334 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:41:07.988303 2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-6381cdd6-6ddf876f9f-8bsrh" Jul 13 22:41:13.025093 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:41:13.025059 2583 generic.go:356] "Generic (PLEG): container finished" podID="618c7b21-2285-401a-b8a4-18dd32b98671" containerID="aacdec7b0e136ef4d9016b5844f8ddd94809f7946a0110ab6b98c7b80ed88b16" exitCode=0 Jul 13 22:41:13.025511 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:41:13.025132 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-6381cdd6-openshift-default-7487bfbcf5-vxwds" event={"ID":"618c7b21-2285-401a-b8a4-18dd32b98671","Type":"ContainerDied","Data":"aacdec7b0e136ef4d9016b5844f8ddd94809f7946a0110ab6b98c7b80ed88b16"} Jul 13 22:41:13.025554 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:41:13.025515 2583 scope.go:117] "RemoveContainer" containerID="aacdec7b0e136ef4d9016b5844f8ddd94809f7946a0110ab6b98c7b80ed88b16" Jul 13 22:41:13.026451 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:41:13.026425 2583 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 13 22:41:13.026503 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:41:13.026491 2583 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 13 22:41:13.026545 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:41:13.026525 2583 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 13 22:41:13.963769 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:41:13.963710 2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-6381cdd6-openshift-default-7487bfbcf5-vxwds" Jul 13 22:41:14.033645 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:41:14.033601 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-6381cdd6-openshift-default-7487bfbcf5-vxwds" event={"ID":"618c7b21-2285-401a-b8a4-18dd32b98671","Type":"ContainerStarted","Data":"07049c43ec8b5f7b10a875eead9d7e81f36189411d66334fb656d7f492fd3189"} Jul 13 22:41:14.177012 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:41:14.176968 2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-6381cdd6-openshift-default-7487bfbcf5-vxwds" Jul 13 22:41:14.178219 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:41:14.178194 2583 patch_prober.go:28] interesting pod/e2e-mt-6381cdd6-openshift-default-7487bfbcf5-vxwds container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.31:15021/healthz/ready\": dial tcp 10.133.0.31:15021: connect: connection refused" start-of-body= Jul 13 22:41:14.178318 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:41:14.178264 2583 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-6381cdd6-openshift-default-7487bfbcf5-vxwds" podUID="618c7b21-2285-401a-b8a4-18dd32b98671" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.31:15021/healthz/ready\": dial tcp 10.133.0.31:15021: connect: connection refused" Jul 13 22:41:15.177091 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:41:15.177047 2583 patch_prober.go:28] interesting pod/e2e-mt-6381cdd6-openshift-default-7487bfbcf5-vxwds container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.31:15021/healthz/ready\": dial tcp 10.133.0.31:15021: connect: connection refused" start-of-body= Jul 13 22:41:15.177530 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:41:15.177131 2583 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-6381cdd6-openshift-default-7487bfbcf5-vxwds" podUID="618c7b21-2285-401a-b8a4-18dd32b98671" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.31:15021/healthz/ready\": dial tcp 10.133.0.31:15021: connect: connection refused" Jul 13 22:41:16.176647 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:41:16.176607 2583 patch_prober.go:28] interesting pod/e2e-mt-6381cdd6-openshift-default-7487bfbcf5-vxwds container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.31:15021/healthz/ready\": dial tcp 10.133.0.31:15021: connect: connection refused" start-of-body= Jul 13 22:41:16.176833 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:41:16.176669 2583 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-6381cdd6-openshift-default-7487bfbcf5-vxwds" podUID="618c7b21-2285-401a-b8a4-18dd32b98671" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.31:15021/healthz/ready\": dial tcp 10.133.0.31:15021: connect: connection refused" Jul 13 22:41:17.176233 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:41:17.176194 2583 patch_prober.go:28] interesting pod/e2e-mt-6381cdd6-openshift-default-7487bfbcf5-vxwds container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.31:15021/healthz/ready\": dial tcp 10.133.0.31:15021: connect: connection refused" start-of-body= Jul 13 22:41:17.176677 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:41:17.176262 2583 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-6381cdd6-openshift-default-7487bfbcf5-vxwds" podUID="618c7b21-2285-401a-b8a4-18dd32b98671" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.31:15021/healthz/ready\": dial tcp 10.133.0.31:15021: connect: connection refused" Jul 13 22:41:18.180580 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:41:18.180530 2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-6381cdd6-openshift-default-7487bfbcf5-vxwds" Jul 13 22:41:18.181025 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:41:18.181003 2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-6381cdd6-openshift-default-7487bfbcf5-vxwds" Jul 13 22:41:18.181644 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:41:18.181626 2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-6381cdd6-openshift-default-7487bfbcf5-vxwds" Jul 13 22:42:02.838320 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:42:02.838288 2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-6381cdd6-6ddf876f9f-8bsrh"] Jul 13 22:42:02.838894 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:42:02.838501 2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-mt-6381cdd6-6ddf876f9f-8bsrh" podUID="1faef928-506a-43e0-97af-bb6ce8b9fda0" containerName="maas-api" containerID="cri-o://ba6cf0cb9e044d3898c9d85a3942c6b79037b7da24182685552d892266e477ea" gracePeriod=30 Jul 13 22:42:02.983532 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:42:02.983479 2583 prober.go:120] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-e2e-mt-6381cdd6-6ddf876f9f-8bsrh" podUID="1faef928-506a-43e0-97af-bb6ce8b9fda0" containerName="maas-api" probeResult="failure" output="Get \"https://10.133.0.32:8443/health\": dial tcp 10.133.0.32:8443: connect: connection refused" Jul 13 22:42:03.090227 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:42:03.090159 2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-6381cdd6-6ddf876f9f-8bsrh" Jul 13 22:42:03.158215 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:42:03.158174 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5w7jk\" (UniqueName: \"kubernetes.io/projected/1faef928-506a-43e0-97af-bb6ce8b9fda0-kube-api-access-5w7jk\") pod \"1faef928-506a-43e0-97af-bb6ce8b9fda0\" (UID: \"1faef928-506a-43e0-97af-bb6ce8b9fda0\") " Jul 13 22:42:03.158428 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:42:03.158225 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1faef928-506a-43e0-97af-bb6ce8b9fda0-maas-api-tls\") pod \"1faef928-506a-43e0-97af-bb6ce8b9fda0\" (UID: \"1faef928-506a-43e0-97af-bb6ce8b9fda0\") " Jul 13 22:42:03.158428 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:42:03.158296 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1faef928-506a-43e0-97af-bb6ce8b9fda0-openshift-service-ca\") pod \"1faef928-506a-43e0-97af-bb6ce8b9fda0\" (UID: \"1faef928-506a-43e0-97af-bb6ce8b9fda0\") " Jul 13 22:42:03.158723 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:42:03.158697 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1faef928-506a-43e0-97af-bb6ce8b9fda0-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "1faef928-506a-43e0-97af-bb6ce8b9fda0" (UID: "1faef928-506a-43e0-97af-bb6ce8b9fda0"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 13 22:42:03.160402 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:42:03.160375 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1faef928-506a-43e0-97af-bb6ce8b9fda0-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "1faef928-506a-43e0-97af-bb6ce8b9fda0" (UID: "1faef928-506a-43e0-97af-bb6ce8b9fda0"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 13 22:42:03.160497 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:42:03.160476 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1faef928-506a-43e0-97af-bb6ce8b9fda0-kube-api-access-5w7jk" (OuterVolumeSpecName: "kube-api-access-5w7jk") pod "1faef928-506a-43e0-97af-bb6ce8b9fda0" (UID: "1faef928-506a-43e0-97af-bb6ce8b9fda0"). InnerVolumeSpecName "kube-api-access-5w7jk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 13 22:42:03.259378 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:42:03.259325 2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5w7jk\" (UniqueName: \"kubernetes.io/projected/1faef928-506a-43e0-97af-bb6ce8b9fda0-kube-api-access-5w7jk\") on node \"ip-10-0-133-179.ec2.internal\" DevicePath \"\"" Jul 13 22:42:03.259378 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:42:03.259371 2583 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1faef928-506a-43e0-97af-bb6ce8b9fda0-maas-api-tls\") on node \"ip-10-0-133-179.ec2.internal\" DevicePath \"\"" Jul 13 22:42:03.259378 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:42:03.259382 2583 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1faef928-506a-43e0-97af-bb6ce8b9fda0-openshift-service-ca\") on node \"ip-10-0-133-179.ec2.internal\" DevicePath \"\"" Jul 13 22:42:03.264728 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:42:03.264699 2583 generic.go:356] "Generic (PLEG): container finished" podID="1faef928-506a-43e0-97af-bb6ce8b9fda0" containerID="ba6cf0cb9e044d3898c9d85a3942c6b79037b7da24182685552d892266e477ea" exitCode=0 Jul 13 22:42:03.264884 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:42:03.264745 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-6381cdd6-6ddf876f9f-8bsrh" event={"ID":"1faef928-506a-43e0-97af-bb6ce8b9fda0","Type":"ContainerDied","Data":"ba6cf0cb9e044d3898c9d85a3942c6b79037b7da24182685552d892266e477ea"} Jul 13 22:42:03.264884 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:42:03.264749 2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-6381cdd6-6ddf876f9f-8bsrh" Jul 13 22:42:03.264884 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:42:03.264788 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-6381cdd6-6ddf876f9f-8bsrh" event={"ID":"1faef928-506a-43e0-97af-bb6ce8b9fda0","Type":"ContainerDied","Data":"8ded0249b86270851acc61c505b957a7521993597064142b1ae59a34051a544e"} Jul 13 22:42:03.264884 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:42:03.264809 2583 scope.go:117] "RemoveContainer" containerID="ba6cf0cb9e044d3898c9d85a3942c6b79037b7da24182685552d892266e477ea" Jul 13 22:42:03.272891 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:42:03.272869 2583 scope.go:117] "RemoveContainer" containerID="ba6cf0cb9e044d3898c9d85a3942c6b79037b7da24182685552d892266e477ea" Jul 13 22:42:03.273204 ip-10-0-133-179 kubenswrapper[2583]: E0713 22:42:03.273179 2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ba6cf0cb9e044d3898c9d85a3942c6b79037b7da24182685552d892266e477ea\": container with ID starting with ba6cf0cb9e044d3898c9d85a3942c6b79037b7da24182685552d892266e477ea not found: ID does not exist" containerID="ba6cf0cb9e044d3898c9d85a3942c6b79037b7da24182685552d892266e477ea" Jul 13 22:42:03.273256 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:42:03.273215 2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ba6cf0cb9e044d3898c9d85a3942c6b79037b7da24182685552d892266e477ea"} err="failed to get container status \"ba6cf0cb9e044d3898c9d85a3942c6b79037b7da24182685552d892266e477ea\": rpc error: code = NotFound desc = could not find container \"ba6cf0cb9e044d3898c9d85a3942c6b79037b7da24182685552d892266e477ea\": container with ID starting with ba6cf0cb9e044d3898c9d85a3942c6b79037b7da24182685552d892266e477ea not found: ID does not exist" Jul 13 22:42:03.284971 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:42:03.284943 2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-6381cdd6-6ddf876f9f-8bsrh"] Jul 13 22:42:03.289190 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:42:03.289165 2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-6381cdd6-6ddf876f9f-8bsrh"] Jul 13 22:42:03.389500 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:42:03.389413 2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1faef928-506a-43e0-97af-bb6ce8b9fda0" path="/var/lib/kubelet/pods/1faef928-506a-43e0-97af-bb6ce8b9fda0/volumes" Jul 13 22:42:56.368144 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:42:56.368101 2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-6381cdd6-openshift-default-7487bfbcf5-vxwds"] Jul 13 22:42:56.368738 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:42:56.368342 2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-6381cdd6-openshift-default-7487bfbcf5-vxwds" podUID="618c7b21-2285-401a-b8a4-18dd32b98671" containerName="istio-proxy" containerID="cri-o://07049c43ec8b5f7b10a875eead9d7e81f36189411d66334fb656d7f492fd3189" gracePeriod=30 Jul 13 22:43:01.517303 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:43:01.517266 2583 generic.go:356] "Generic (PLEG): container finished" podID="618c7b21-2285-401a-b8a4-18dd32b98671" containerID="07049c43ec8b5f7b10a875eead9d7e81f36189411d66334fb656d7f492fd3189" exitCode=0 Jul 13 22:43:01.517791 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:43:01.517339 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-6381cdd6-openshift-default-7487bfbcf5-vxwds" event={"ID":"618c7b21-2285-401a-b8a4-18dd32b98671","Type":"ContainerDied","Data":"07049c43ec8b5f7b10a875eead9d7e81f36189411d66334fb656d7f492fd3189"} Jul 13 22:43:01.517791 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:43:01.517418 2583 scope.go:117] "RemoveContainer" containerID="aacdec7b0e136ef4d9016b5844f8ddd94809f7946a0110ab6b98c7b80ed88b16" Jul 13 22:43:01.610881 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:43:01.610856 2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-6381cdd6-openshift-default-7487bfbcf5-vxwds" Jul 13 22:43:01.757244 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:43:01.757147 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/618c7b21-2285-401a-b8a4-18dd32b98671-credential-socket\") pod \"618c7b21-2285-401a-b8a4-18dd32b98671\" (UID: \"618c7b21-2285-401a-b8a4-18dd32b98671\") " Jul 13 22:43:01.757244 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:43:01.757190 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/618c7b21-2285-401a-b8a4-18dd32b98671-istiod-ca-cert\") pod \"618c7b21-2285-401a-b8a4-18dd32b98671\" (UID: \"618c7b21-2285-401a-b8a4-18dd32b98671\") " Jul 13 22:43:01.757244 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:43:01.757209 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/618c7b21-2285-401a-b8a4-18dd32b98671-istio-token\") pod \"618c7b21-2285-401a-b8a4-18dd32b98671\" (UID: \"618c7b21-2285-401a-b8a4-18dd32b98671\") " Jul 13 22:43:01.757244 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:43:01.757240 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/618c7b21-2285-401a-b8a4-18dd32b98671-istio-podinfo\") pod \"618c7b21-2285-401a-b8a4-18dd32b98671\" (UID: \"618c7b21-2285-401a-b8a4-18dd32b98671\") " Jul 13 22:43:01.757559 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:43:01.757305 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/618c7b21-2285-401a-b8a4-18dd32b98671-istio-data\") pod \"618c7b21-2285-401a-b8a4-18dd32b98671\" (UID: \"618c7b21-2285-401a-b8a4-18dd32b98671\") " Jul 13 22:43:01.757559 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:43:01.757336 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-pvrw5\" (UniqueName: \"kubernetes.io/projected/618c7b21-2285-401a-b8a4-18dd32b98671-kube-api-access-pvrw5\") pod \"618c7b21-2285-401a-b8a4-18dd32b98671\" (UID: \"618c7b21-2285-401a-b8a4-18dd32b98671\") " Jul 13 22:43:01.757559 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:43:01.757359 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/618c7b21-2285-401a-b8a4-18dd32b98671-workload-socket\") pod \"618c7b21-2285-401a-b8a4-18dd32b98671\" (UID: \"618c7b21-2285-401a-b8a4-18dd32b98671\") " Jul 13 22:43:01.757559 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:43:01.757411 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/618c7b21-2285-401a-b8a4-18dd32b98671-istio-envoy\") pod \"618c7b21-2285-401a-b8a4-18dd32b98671\" (UID: \"618c7b21-2285-401a-b8a4-18dd32b98671\") " Jul 13 22:43:01.757559 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:43:01.757435 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/618c7b21-2285-401a-b8a4-18dd32b98671-workload-certs\") pod \"618c7b21-2285-401a-b8a4-18dd32b98671\" (UID: \"618c7b21-2285-401a-b8a4-18dd32b98671\") " Jul 13 22:43:01.757559 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:43:01.757484 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/618c7b21-2285-401a-b8a4-18dd32b98671-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "618c7b21-2285-401a-b8a4-18dd32b98671" (UID: "618c7b21-2285-401a-b8a4-18dd32b98671"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 13 22:43:01.757903 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:43:01.757728 2583 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/618c7b21-2285-401a-b8a4-18dd32b98671-credential-socket\") on node \"ip-10-0-133-179.ec2.internal\" DevicePath \"\"" Jul 13 22:43:01.757903 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:43:01.757738 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/618c7b21-2285-401a-b8a4-18dd32b98671-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "618c7b21-2285-401a-b8a4-18dd32b98671" (UID: "618c7b21-2285-401a-b8a4-18dd32b98671"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 13 22:43:01.757903 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:43:01.757751 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/618c7b21-2285-401a-b8a4-18dd32b98671-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "618c7b21-2285-401a-b8a4-18dd32b98671" (UID: "618c7b21-2285-401a-b8a4-18dd32b98671"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 13 22:43:01.757903 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:43:01.757777 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/618c7b21-2285-401a-b8a4-18dd32b98671-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "618c7b21-2285-401a-b8a4-18dd32b98671" (UID: "618c7b21-2285-401a-b8a4-18dd32b98671"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 13 22:43:01.758285 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:43:01.758264 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/618c7b21-2285-401a-b8a4-18dd32b98671-istio-data" (OuterVolumeSpecName: "istio-data") pod "618c7b21-2285-401a-b8a4-18dd32b98671" (UID: "618c7b21-2285-401a-b8a4-18dd32b98671"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 13 22:43:01.759521 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:43:01.759500 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/618c7b21-2285-401a-b8a4-18dd32b98671-istio-token" (OuterVolumeSpecName: "istio-token") pod "618c7b21-2285-401a-b8a4-18dd32b98671" (UID: "618c7b21-2285-401a-b8a4-18dd32b98671"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 13 22:43:01.759707 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:43:01.759690 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/618c7b21-2285-401a-b8a4-18dd32b98671-kube-api-access-pvrw5" (OuterVolumeSpecName: "kube-api-access-pvrw5") pod "618c7b21-2285-401a-b8a4-18dd32b98671" (UID: "618c7b21-2285-401a-b8a4-18dd32b98671"). InnerVolumeSpecName "kube-api-access-pvrw5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 13 22:43:01.759915 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:43:01.759894 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/618c7b21-2285-401a-b8a4-18dd32b98671-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "618c7b21-2285-401a-b8a4-18dd32b98671" (UID: "618c7b21-2285-401a-b8a4-18dd32b98671"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 13 22:43:01.759915 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:43:01.759901 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/618c7b21-2285-401a-b8a4-18dd32b98671-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "618c7b21-2285-401a-b8a4-18dd32b98671" (UID: "618c7b21-2285-401a-b8a4-18dd32b98671"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 13 22:43:01.858519 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:43:01.858480 2583 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/618c7b21-2285-401a-b8a4-18dd32b98671-istio-envoy\") on node \"ip-10-0-133-179.ec2.internal\" DevicePath \"\"" Jul 13 22:43:01.858519 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:43:01.858513 2583 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/618c7b21-2285-401a-b8a4-18dd32b98671-workload-certs\") on node \"ip-10-0-133-179.ec2.internal\" DevicePath \"\"" Jul 13 22:43:01.858519 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:43:01.858522 2583 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/618c7b21-2285-401a-b8a4-18dd32b98671-istiod-ca-cert\") on node \"ip-10-0-133-179.ec2.internal\" DevicePath \"\"" Jul 13 22:43:01.858519 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:43:01.858529 2583 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/618c7b21-2285-401a-b8a4-18dd32b98671-istio-token\") on node \"ip-10-0-133-179.ec2.internal\" DevicePath \"\"" Jul 13 22:43:01.858812 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:43:01.858538 2583 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/618c7b21-2285-401a-b8a4-18dd32b98671-istio-podinfo\") on node \"ip-10-0-133-179.ec2.internal\" DevicePath \"\"" Jul 13 22:43:01.858812 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:43:01.858546 2583 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/618c7b21-2285-401a-b8a4-18dd32b98671-istio-data\") on node \"ip-10-0-133-179.ec2.internal\" DevicePath \"\"" Jul 13 22:43:01.858812 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:43:01.858554 2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pvrw5\" (UniqueName: \"kubernetes.io/projected/618c7b21-2285-401a-b8a4-18dd32b98671-kube-api-access-pvrw5\") on node \"ip-10-0-133-179.ec2.internal\" DevicePath \"\"" Jul 13 22:43:01.858812 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:43:01.858562 2583 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/618c7b21-2285-401a-b8a4-18dd32b98671-workload-socket\") on node \"ip-10-0-133-179.ec2.internal\" DevicePath \"\"" Jul 13 22:43:02.523876 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:43:02.523848 2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-6381cdd6-openshift-default-7487bfbcf5-vxwds" Jul 13 22:43:02.524309 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:43:02.523849 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-6381cdd6-openshift-default-7487bfbcf5-vxwds" event={"ID":"618c7b21-2285-401a-b8a4-18dd32b98671","Type":"ContainerDied","Data":"9cb248e0e838d9f8dfe197301b6064de26069893c2757e692cf52830e3bc6b2e"} Jul 13 22:43:02.524309 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:43:02.523982 2583 scope.go:117] "RemoveContainer" containerID="07049c43ec8b5f7b10a875eead9d7e81f36189411d66334fb656d7f492fd3189" Jul 13 22:43:02.546538 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:43:02.546508 2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-6381cdd6-openshift-default-7487bfbcf5-vxwds"] Jul 13 22:43:02.552076 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:43:02.552052 2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-6381cdd6-openshift-default-7487bfbcf5-vxwds"] Jul 13 22:43:03.389471 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:43:03.389435 2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="618c7b21-2285-401a-b8a4-18dd32b98671" path="/var/lib/kubelet/pods/618c7b21-2285-401a-b8a4-18dd32b98671/volumes" Jul 13 22:43:50.828713 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:43:50.828678 2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-6e7f3c-openshift-default-5f488c48df-w2rvd"] Jul 13 22:43:50.829244 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:43:50.828974 2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="618c7b21-2285-401a-b8a4-18dd32b98671" containerName="istio-proxy" Jul 13 22:43:50.829244 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:43:50.828984 2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="618c7b21-2285-401a-b8a4-18dd32b98671" containerName="istio-proxy" Jul 13 22:43:50.829244 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:43:50.828997 2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="618c7b21-2285-401a-b8a4-18dd32b98671" containerName="istio-proxy" Jul 13 22:43:50.829244 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:43:50.829003 2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="618c7b21-2285-401a-b8a4-18dd32b98671" containerName="istio-proxy" Jul 13 22:43:50.829244 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:43:50.829013 2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1faef928-506a-43e0-97af-bb6ce8b9fda0" containerName="maas-api" Jul 13 22:43:50.829244 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:43:50.829019 2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="1faef928-506a-43e0-97af-bb6ce8b9fda0" containerName="maas-api" Jul 13 22:43:50.829244 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:43:50.829066 2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="618c7b21-2285-401a-b8a4-18dd32b98671" containerName="istio-proxy" Jul 13 22:43:50.829244 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:43:50.829074 2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="618c7b21-2285-401a-b8a4-18dd32b98671" containerName="istio-proxy" Jul 13 22:43:50.829244 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:43:50.829082 2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="1faef928-506a-43e0-97af-bb6ce8b9fda0" containerName="maas-api" Jul 13 22:43:50.831236 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:43:50.831215 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-6e7f3c-openshift-default-5f488c48df-w2rvd" Jul 13 22:43:50.834834 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:43:50.834811 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-a-6e7f3c-openshift-default-dockercfg-lm6m6\"" Jul 13 22:43:50.846138 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:43:50.846108 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-6e7f3c-openshift-default-5f488c48df-w2rvd"] Jul 13 22:43:50.939284 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:43:50.939246 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7bafdacd-97b2-440e-ab73-a3591de9b949-credential-socket\") pod \"e2e-inf-a-6e7f3c-openshift-default-5f488c48df-w2rvd\" (UID: \"7bafdacd-97b2-440e-ab73-a3591de9b949\") " pod="openshift-ingress/e2e-inf-a-6e7f3c-openshift-default-5f488c48df-w2rvd" Jul 13 22:43:50.939284 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:43:50.939291 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7bafdacd-97b2-440e-ab73-a3591de9b949-istio-envoy\") pod \"e2e-inf-a-6e7f3c-openshift-default-5f488c48df-w2rvd\" (UID: \"7bafdacd-97b2-440e-ab73-a3591de9b949\") " pod="openshift-ingress/e2e-inf-a-6e7f3c-openshift-default-5f488c48df-w2rvd" Jul 13 22:43:50.939556 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:43:50.939356 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7bafdacd-97b2-440e-ab73-a3591de9b949-istio-data\") pod \"e2e-inf-a-6e7f3c-openshift-default-5f488c48df-w2rvd\" (UID: \"7bafdacd-97b2-440e-ab73-a3591de9b949\") " pod="openshift-ingress/e2e-inf-a-6e7f3c-openshift-default-5f488c48df-w2rvd" Jul 13 22:43:50.939556 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:43:50.939394 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7bafdacd-97b2-440e-ab73-a3591de9b949-istio-token\") pod \"e2e-inf-a-6e7f3c-openshift-default-5f488c48df-w2rvd\" (UID: \"7bafdacd-97b2-440e-ab73-a3591de9b949\") " pod="openshift-ingress/e2e-inf-a-6e7f3c-openshift-default-5f488c48df-w2rvd" Jul 13 22:43:50.939556 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:43:50.939487 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7bafdacd-97b2-440e-ab73-a3591de9b949-istiod-ca-cert\") pod \"e2e-inf-a-6e7f3c-openshift-default-5f488c48df-w2rvd\" (UID: \"7bafdacd-97b2-440e-ab73-a3591de9b949\") " pod="openshift-ingress/e2e-inf-a-6e7f3c-openshift-default-5f488c48df-w2rvd" Jul 13 22:43:50.939556 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:43:50.939518 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7bafdacd-97b2-440e-ab73-a3591de9b949-workload-certs\") pod \"e2e-inf-a-6e7f3c-openshift-default-5f488c48df-w2rvd\" (UID: \"7bafdacd-97b2-440e-ab73-a3591de9b949\") " pod="openshift-ingress/e2e-inf-a-6e7f3c-openshift-default-5f488c48df-w2rvd" Jul 13 22:43:50.939556 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:43:50.939547 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7bafdacd-97b2-440e-ab73-a3591de9b949-istio-podinfo\") pod \"e2e-inf-a-6e7f3c-openshift-default-5f488c48df-w2rvd\" (UID: \"7bafdacd-97b2-440e-ab73-a3591de9b949\") " pod="openshift-ingress/e2e-inf-a-6e7f3c-openshift-default-5f488c48df-w2rvd" Jul 13 22:43:50.939847 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:43:50.939592 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7bafdacd-97b2-440e-ab73-a3591de9b949-workload-socket\") pod \"e2e-inf-a-6e7f3c-openshift-default-5f488c48df-w2rvd\" (UID: \"7bafdacd-97b2-440e-ab73-a3591de9b949\") " pod="openshift-ingress/e2e-inf-a-6e7f3c-openshift-default-5f488c48df-w2rvd" Jul 13 22:43:50.939847 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:43:50.939627 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dkj92\" (UniqueName: \"kubernetes.io/projected/7bafdacd-97b2-440e-ab73-a3591de9b949-kube-api-access-dkj92\") pod \"e2e-inf-a-6e7f3c-openshift-default-5f488c48df-w2rvd\" (UID: \"7bafdacd-97b2-440e-ab73-a3591de9b949\") " pod="openshift-ingress/e2e-inf-a-6e7f3c-openshift-default-5f488c48df-w2rvd" Jul 13 22:43:51.040427 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:43:51.040384 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7bafdacd-97b2-440e-ab73-a3591de9b949-istiod-ca-cert\") pod \"e2e-inf-a-6e7f3c-openshift-default-5f488c48df-w2rvd\" (UID: \"7bafdacd-97b2-440e-ab73-a3591de9b949\") " pod="openshift-ingress/e2e-inf-a-6e7f3c-openshift-default-5f488c48df-w2rvd" Jul 13 22:43:51.040427 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:43:51.040427 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7bafdacd-97b2-440e-ab73-a3591de9b949-workload-certs\") pod \"e2e-inf-a-6e7f3c-openshift-default-5f488c48df-w2rvd\" (UID: \"7bafdacd-97b2-440e-ab73-a3591de9b949\") " pod="openshift-ingress/e2e-inf-a-6e7f3c-openshift-default-5f488c48df-w2rvd" Jul 13 22:43:51.040718 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:43:51.040464 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7bafdacd-97b2-440e-ab73-a3591de9b949-istio-podinfo\") pod \"e2e-inf-a-6e7f3c-openshift-default-5f488c48df-w2rvd\" (UID: \"7bafdacd-97b2-440e-ab73-a3591de9b949\") " pod="openshift-ingress/e2e-inf-a-6e7f3c-openshift-default-5f488c48df-w2rvd" Jul 13 22:43:51.040718 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:43:51.040503 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7bafdacd-97b2-440e-ab73-a3591de9b949-workload-socket\") pod \"e2e-inf-a-6e7f3c-openshift-default-5f488c48df-w2rvd\" (UID: \"7bafdacd-97b2-440e-ab73-a3591de9b949\") " pod="openshift-ingress/e2e-inf-a-6e7f3c-openshift-default-5f488c48df-w2rvd" Jul 13 22:43:51.040718 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:43:51.040653 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dkj92\" (UniqueName: \"kubernetes.io/projected/7bafdacd-97b2-440e-ab73-a3591de9b949-kube-api-access-dkj92\") pod \"e2e-inf-a-6e7f3c-openshift-default-5f488c48df-w2rvd\" (UID: \"7bafdacd-97b2-440e-ab73-a3591de9b949\") " pod="openshift-ingress/e2e-inf-a-6e7f3c-openshift-default-5f488c48df-w2rvd" Jul 13 22:43:51.040718 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:43:51.040713 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7bafdacd-97b2-440e-ab73-a3591de9b949-credential-socket\") pod \"e2e-inf-a-6e7f3c-openshift-default-5f488c48df-w2rvd\" (UID: \"7bafdacd-97b2-440e-ab73-a3591de9b949\") " pod="openshift-ingress/e2e-inf-a-6e7f3c-openshift-default-5f488c48df-w2rvd" Jul 13 22:43:51.040915 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:43:51.040754 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7bafdacd-97b2-440e-ab73-a3591de9b949-istio-envoy\") pod \"e2e-inf-a-6e7f3c-openshift-default-5f488c48df-w2rvd\" (UID: \"7bafdacd-97b2-440e-ab73-a3591de9b949\") " pod="openshift-ingress/e2e-inf-a-6e7f3c-openshift-default-5f488c48df-w2rvd" Jul 13 22:43:51.040915 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:43:51.040789 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7bafdacd-97b2-440e-ab73-a3591de9b949-istio-data\") pod \"e2e-inf-a-6e7f3c-openshift-default-5f488c48df-w2rvd\" (UID: \"7bafdacd-97b2-440e-ab73-a3591de9b949\") " pod="openshift-ingress/e2e-inf-a-6e7f3c-openshift-default-5f488c48df-w2rvd" Jul 13 22:43:51.040915 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:43:51.040820 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7bafdacd-97b2-440e-ab73-a3591de9b949-istio-token\") pod \"e2e-inf-a-6e7f3c-openshift-default-5f488c48df-w2rvd\" (UID: \"7bafdacd-97b2-440e-ab73-a3591de9b949\") " pod="openshift-ingress/e2e-inf-a-6e7f3c-openshift-default-5f488c48df-w2rvd" Jul 13 22:43:51.041059 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:43:51.041029 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7bafdacd-97b2-440e-ab73-a3591de9b949-workload-socket\") pod \"e2e-inf-a-6e7f3c-openshift-default-5f488c48df-w2rvd\" (UID: \"7bafdacd-97b2-440e-ab73-a3591de9b949\") " pod="openshift-ingress/e2e-inf-a-6e7f3c-openshift-default-5f488c48df-w2rvd" Jul 13 22:43:51.041111 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:43:51.041076 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7bafdacd-97b2-440e-ab73-a3591de9b949-credential-socket\") pod \"e2e-inf-a-6e7f3c-openshift-default-5f488c48df-w2rvd\" (UID: \"7bafdacd-97b2-440e-ab73-a3591de9b949\") " pod="openshift-ingress/e2e-inf-a-6e7f3c-openshift-default-5f488c48df-w2rvd" Jul 13 22:43:51.041111 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:43:51.041096 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7bafdacd-97b2-440e-ab73-a3591de9b949-workload-certs\") pod \"e2e-inf-a-6e7f3c-openshift-default-5f488c48df-w2rvd\" (UID: \"7bafdacd-97b2-440e-ab73-a3591de9b949\") " pod="openshift-ingress/e2e-inf-a-6e7f3c-openshift-default-5f488c48df-w2rvd" Jul 13 22:43:51.041319 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:43:51.041297 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7bafdacd-97b2-440e-ab73-a3591de9b949-istiod-ca-cert\") pod \"e2e-inf-a-6e7f3c-openshift-default-5f488c48df-w2rvd\" (UID: \"7bafdacd-97b2-440e-ab73-a3591de9b949\") " pod="openshift-ingress/e2e-inf-a-6e7f3c-openshift-default-5f488c48df-w2rvd" Jul 13 22:43:51.041319 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:43:51.041313 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7bafdacd-97b2-440e-ab73-a3591de9b949-istio-data\") pod \"e2e-inf-a-6e7f3c-openshift-default-5f488c48df-w2rvd\" (UID: \"7bafdacd-97b2-440e-ab73-a3591de9b949\") " pod="openshift-ingress/e2e-inf-a-6e7f3c-openshift-default-5f488c48df-w2rvd" Jul 13 22:43:51.043078 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:43:51.043058 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7bafdacd-97b2-440e-ab73-a3591de9b949-istio-envoy\") pod \"e2e-inf-a-6e7f3c-openshift-default-5f488c48df-w2rvd\" (UID: \"7bafdacd-97b2-440e-ab73-a3591de9b949\") " pod="openshift-ingress/e2e-inf-a-6e7f3c-openshift-default-5f488c48df-w2rvd" Jul 13 22:43:51.043226 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:43:51.043209 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7bafdacd-97b2-440e-ab73-a3591de9b949-istio-podinfo\") pod \"e2e-inf-a-6e7f3c-openshift-default-5f488c48df-w2rvd\" (UID: \"7bafdacd-97b2-440e-ab73-a3591de9b949\") " pod="openshift-ingress/e2e-inf-a-6e7f3c-openshift-default-5f488c48df-w2rvd" Jul 13 22:43:51.048368 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:43:51.048337 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7bafdacd-97b2-440e-ab73-a3591de9b949-istio-token\") pod \"e2e-inf-a-6e7f3c-openshift-default-5f488c48df-w2rvd\" (UID: \"7bafdacd-97b2-440e-ab73-a3591de9b949\") " pod="openshift-ingress/e2e-inf-a-6e7f3c-openshift-default-5f488c48df-w2rvd" Jul 13 22:43:51.048543 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:43:51.048524 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dkj92\" (UniqueName: \"kubernetes.io/projected/7bafdacd-97b2-440e-ab73-a3591de9b949-kube-api-access-dkj92\") pod \"e2e-inf-a-6e7f3c-openshift-default-5f488c48df-w2rvd\" (UID: \"7bafdacd-97b2-440e-ab73-a3591de9b949\") " pod="openshift-ingress/e2e-inf-a-6e7f3c-openshift-default-5f488c48df-w2rvd" Jul 13 22:43:51.141191 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:43:51.141086 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-6e7f3c-openshift-default-5f488c48df-w2rvd" Jul 13 22:43:51.277735 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:43:51.277709 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-6e7f3c-openshift-default-5f488c48df-w2rvd"] Jul 13 22:43:51.279980 ip-10-0-133-179 kubenswrapper[2583]: W0713 22:43:51.279949 2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7bafdacd_97b2_440e_ab73_a3591de9b949.slice/crio-ddd22d904cd1ef9871fd7ebb053668d13c054486b867c0ed462e381c476d8d1b WatchSource:0}: Error finding container ddd22d904cd1ef9871fd7ebb053668d13c054486b867c0ed462e381c476d8d1b: Status 404 returned error can't find the container with id ddd22d904cd1ef9871fd7ebb053668d13c054486b867c0ed462e381c476d8d1b Jul 13 22:43:51.282098 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:43:51.282064 2583 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 13 22:43:51.282184 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:43:51.282140 2583 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 13 22:43:51.282222 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:43:51.282189 2583 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 13 22:43:51.721177 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:43:51.721143 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-6e7f3c-openshift-default-5f488c48df-w2rvd" event={"ID":"7bafdacd-97b2-440e-ab73-a3591de9b949","Type":"ContainerStarted","Data":"9bfb85ecfd065b23dd78f95d5591e76bfc49c3f6ab59556885a32181a9863f4f"} Jul 13 22:43:51.721391 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:43:51.721195 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-6e7f3c-openshift-default-5f488c48df-w2rvd" event={"ID":"7bafdacd-97b2-440e-ab73-a3591de9b949","Type":"ContainerStarted","Data":"ddd22d904cd1ef9871fd7ebb053668d13c054486b867c0ed462e381c476d8d1b"} Jul 13 22:43:51.741092 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:43:51.741041 2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-6e7f3c-openshift-default-5f488c48df-w2rvd" podStartSLOduration=1.741024599 podStartE2EDuration="1.741024599s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-13 22:43:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 22:43:51.740254622 +0000 UTC m=+1826.988169972" watchObservedRunningTime="2026-07-13 22:43:51.741024599 +0000 UTC m=+1826.988939948" Jul 13 22:43:52.141701 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:43:52.141644 2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-6e7f3c-openshift-default-5f488c48df-w2rvd" Jul 13 22:43:52.146734 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:43:52.146710 2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-6e7f3c-openshift-default-5f488c48df-w2rvd" Jul 13 22:43:52.726014 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:43:52.725979 2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-6e7f3c-openshift-default-5f488c48df-w2rvd" Jul 13 22:43:52.726891 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:43:52.726875 2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-6e7f3c-openshift-default-5f488c48df-w2rvd" Jul 13 22:43:53.755423 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:43:53.755386 2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-6e7f3c-7b4fc9f994-cbddg"] Jul 13 22:43:53.758077 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:43:53.758055 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-6e7f3c-7b4fc9f994-cbddg" Jul 13 22:43:53.760330 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:43:53.760301 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-a-6e7f3c-serving-cert\"" Jul 13 22:43:53.767656 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:43:53.767622 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-6e7f3c-7b4fc9f994-cbddg"] Jul 13 22:43:53.865848 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:43:53.865805 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pscxq\" (UniqueName: \"kubernetes.io/projected/f13fb69b-5c84-43b2-8606-0607ab353004-kube-api-access-pscxq\") pod \"maas-api-e2e-inf-a-6e7f3c-7b4fc9f994-cbddg\" (UID: \"f13fb69b-5c84-43b2-8606-0607ab353004\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-6e7f3c-7b4fc9f994-cbddg" Jul 13 22:43:53.866043 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:43:53.865859 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f13fb69b-5c84-43b2-8606-0607ab353004-maas-api-tls\") pod \"maas-api-e2e-inf-a-6e7f3c-7b4fc9f994-cbddg\" (UID: \"f13fb69b-5c84-43b2-8606-0607ab353004\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-6e7f3c-7b4fc9f994-cbddg" Jul 13 22:43:53.866043 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:43:53.865982 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f13fb69b-5c84-43b2-8606-0607ab353004-openshift-service-ca\") pod \"maas-api-e2e-inf-a-6e7f3c-7b4fc9f994-cbddg\" (UID: \"f13fb69b-5c84-43b2-8606-0607ab353004\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-6e7f3c-7b4fc9f994-cbddg" Jul 13 22:43:53.966996 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:43:53.966960 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pscxq\" (UniqueName: \"kubernetes.io/projected/f13fb69b-5c84-43b2-8606-0607ab353004-kube-api-access-pscxq\") pod \"maas-api-e2e-inf-a-6e7f3c-7b4fc9f994-cbddg\" (UID: \"f13fb69b-5c84-43b2-8606-0607ab353004\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-6e7f3c-7b4fc9f994-cbddg" Jul 13 22:43:53.966996 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:43:53.967003 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f13fb69b-5c84-43b2-8606-0607ab353004-maas-api-tls\") pod \"maas-api-e2e-inf-a-6e7f3c-7b4fc9f994-cbddg\" (UID: \"f13fb69b-5c84-43b2-8606-0607ab353004\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-6e7f3c-7b4fc9f994-cbddg" Jul 13 22:43:53.967216 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:43:53.967046 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f13fb69b-5c84-43b2-8606-0607ab353004-openshift-service-ca\") pod \"maas-api-e2e-inf-a-6e7f3c-7b4fc9f994-cbddg\" (UID: \"f13fb69b-5c84-43b2-8606-0607ab353004\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-6e7f3c-7b4fc9f994-cbddg" Jul 13 22:43:53.967716 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:43:53.967688 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f13fb69b-5c84-43b2-8606-0607ab353004-openshift-service-ca\") pod \"maas-api-e2e-inf-a-6e7f3c-7b4fc9f994-cbddg\" (UID: \"f13fb69b-5c84-43b2-8606-0607ab353004\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-6e7f3c-7b4fc9f994-cbddg" Jul 13 22:43:53.969475 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:43:53.969455 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f13fb69b-5c84-43b2-8606-0607ab353004-maas-api-tls\") pod \"maas-api-e2e-inf-a-6e7f3c-7b4fc9f994-cbddg\" (UID: \"f13fb69b-5c84-43b2-8606-0607ab353004\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-6e7f3c-7b4fc9f994-cbddg" Jul 13 22:43:53.974717 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:43:53.974694 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pscxq\" (UniqueName: \"kubernetes.io/projected/f13fb69b-5c84-43b2-8606-0607ab353004-kube-api-access-pscxq\") pod \"maas-api-e2e-inf-a-6e7f3c-7b4fc9f994-cbddg\" (UID: \"f13fb69b-5c84-43b2-8606-0607ab353004\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-6e7f3c-7b4fc9f994-cbddg" Jul 13 22:43:54.068945 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:43:54.068906 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-6e7f3c-7b4fc9f994-cbddg" Jul 13 22:43:54.207732 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:43:54.207656 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-6e7f3c-7b4fc9f994-cbddg"] Jul 13 22:43:54.210387 ip-10-0-133-179 kubenswrapper[2583]: W0713 22:43:54.210355 2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf13fb69b_5c84_43b2_8606_0607ab353004.slice/crio-38a9d66548624d265ab27f960881d9de5c03df6dac8f0670284d401a9b763ac9 WatchSource:0}: Error finding container 38a9d66548624d265ab27f960881d9de5c03df6dac8f0670284d401a9b763ac9: Status 404 returned error can't find the container with id 38a9d66548624d265ab27f960881d9de5c03df6dac8f0670284d401a9b763ac9 Jul 13 22:43:54.211819 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:43:54.211802 2583 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 13 22:43:54.735797 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:43:54.735763 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-6e7f3c-7b4fc9f994-cbddg" event={"ID":"f13fb69b-5c84-43b2-8606-0607ab353004","Type":"ContainerStarted","Data":"4b35918e5ebfae35362f24642a343bf7b111462e9e026444daa630bcb5565506"} Jul 13 22:43:54.736099 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:43:54.735816 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-6e7f3c-7b4fc9f994-cbddg" event={"ID":"f13fb69b-5c84-43b2-8606-0607ab353004","Type":"ContainerStarted","Data":"38a9d66548624d265ab27f960881d9de5c03df6dac8f0670284d401a9b763ac9"} Jul 13 22:43:54.776982 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:43:54.776930 2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-6e7f3c-7b4fc9f994-cbddg" podStartSLOduration=1.369329761 podStartE2EDuration="1.776914374s" totalImagesPullingTime="407.584613ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-13 22:43:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 22:43:54.774421834 +0000 UTC m=+1830.022337223" watchObservedRunningTime="2026-07-13 22:43:54.776914374 +0000 UTC m=+1830.024829724" Jul 13 22:43:55.740214 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:43:55.740182 2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-6e7f3c-7b4fc9f994-cbddg" Jul 13 22:43:58.798004 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:43:58.797969 2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-b-953e38-openshift-default-5567878d9c-r76gn"] Jul 13 22:43:58.800242 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:43:58.800224 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-953e38-openshift-default-5567878d9c-r76gn" Jul 13 22:43:58.808153 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:43:58.808122 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-b-953e38-openshift-default-dockercfg-bvdh9\"" Jul 13 22:43:58.845953 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:43:58.845917 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-953e38-openshift-default-5567878d9c-r76gn"] Jul 13 22:43:58.911560 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:43:58.911520 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d9c803a7-01f3-409c-857f-48b2246e009f-istiod-ca-cert\") pod \"e2e-inf-b-953e38-openshift-default-5567878d9c-r76gn\" (UID: \"d9c803a7-01f3-409c-857f-48b2246e009f\") " pod="openshift-ingress/e2e-inf-b-953e38-openshift-default-5567878d9c-r76gn" Jul 13 22:43:58.911786 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:43:58.911561 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d9c803a7-01f3-409c-857f-48b2246e009f-credential-socket\") pod \"e2e-inf-b-953e38-openshift-default-5567878d9c-r76gn\" (UID: \"d9c803a7-01f3-409c-857f-48b2246e009f\") " pod="openshift-ingress/e2e-inf-b-953e38-openshift-default-5567878d9c-r76gn" Jul 13 22:43:58.911786 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:43:58.911635 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d9c803a7-01f3-409c-857f-48b2246e009f-istio-envoy\") pod \"e2e-inf-b-953e38-openshift-default-5567878d9c-r76gn\" (UID: \"d9c803a7-01f3-409c-857f-48b2246e009f\") " pod="openshift-ingress/e2e-inf-b-953e38-openshift-default-5567878d9c-r76gn" Jul 13 22:43:58.911786 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:43:58.911694 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d9c803a7-01f3-409c-857f-48b2246e009f-istio-data\") pod \"e2e-inf-b-953e38-openshift-default-5567878d9c-r76gn\" (UID: \"d9c803a7-01f3-409c-857f-48b2246e009f\") " pod="openshift-ingress/e2e-inf-b-953e38-openshift-default-5567878d9c-r76gn" Jul 13 22:43:58.911786 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:43:58.911715 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d9c803a7-01f3-409c-857f-48b2246e009f-istio-token\") pod \"e2e-inf-b-953e38-openshift-default-5567878d9c-r76gn\" (UID: \"d9c803a7-01f3-409c-857f-48b2246e009f\") " pod="openshift-ingress/e2e-inf-b-953e38-openshift-default-5567878d9c-r76gn" Jul 13 22:43:58.912000 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:43:58.911797 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d9c803a7-01f3-409c-857f-48b2246e009f-workload-certs\") pod \"e2e-inf-b-953e38-openshift-default-5567878d9c-r76gn\" (UID: \"d9c803a7-01f3-409c-857f-48b2246e009f\") " pod="openshift-ingress/e2e-inf-b-953e38-openshift-default-5567878d9c-r76gn" Jul 13 22:43:58.912000 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:43:58.911831 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d9c803a7-01f3-409c-857f-48b2246e009f-workload-socket\") pod \"e2e-inf-b-953e38-openshift-default-5567878d9c-r76gn\" (UID: \"d9c803a7-01f3-409c-857f-48b2246e009f\") " pod="openshift-ingress/e2e-inf-b-953e38-openshift-default-5567878d9c-r76gn" Jul 13 22:43:58.912000 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:43:58.911846 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d9c803a7-01f3-409c-857f-48b2246e009f-istio-podinfo\") pod \"e2e-inf-b-953e38-openshift-default-5567878d9c-r76gn\" (UID: \"d9c803a7-01f3-409c-857f-48b2246e009f\") " pod="openshift-ingress/e2e-inf-b-953e38-openshift-default-5567878d9c-r76gn" Jul 13 22:43:58.912000 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:43:58.911869 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2jfvp\" (UniqueName: \"kubernetes.io/projected/d9c803a7-01f3-409c-857f-48b2246e009f-kube-api-access-2jfvp\") pod \"e2e-inf-b-953e38-openshift-default-5567878d9c-r76gn\" (UID: \"d9c803a7-01f3-409c-857f-48b2246e009f\") " pod="openshift-ingress/e2e-inf-b-953e38-openshift-default-5567878d9c-r76gn" Jul 13 22:43:59.012914 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:43:59.012874 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d9c803a7-01f3-409c-857f-48b2246e009f-istiod-ca-cert\") pod \"e2e-inf-b-953e38-openshift-default-5567878d9c-r76gn\" (UID: \"d9c803a7-01f3-409c-857f-48b2246e009f\") " pod="openshift-ingress/e2e-inf-b-953e38-openshift-default-5567878d9c-r76gn" Jul 13 22:43:59.012914 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:43:59.012914 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d9c803a7-01f3-409c-857f-48b2246e009f-credential-socket\") pod \"e2e-inf-b-953e38-openshift-default-5567878d9c-r76gn\" (UID: \"d9c803a7-01f3-409c-857f-48b2246e009f\") " pod="openshift-ingress/e2e-inf-b-953e38-openshift-default-5567878d9c-r76gn" Jul 13 22:43:59.013167 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:43:59.012936 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d9c803a7-01f3-409c-857f-48b2246e009f-istio-envoy\") pod \"e2e-inf-b-953e38-openshift-default-5567878d9c-r76gn\" (UID: \"d9c803a7-01f3-409c-857f-48b2246e009f\") " pod="openshift-ingress/e2e-inf-b-953e38-openshift-default-5567878d9c-r76gn" Jul 13 22:43:59.013167 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:43:59.012960 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d9c803a7-01f3-409c-857f-48b2246e009f-istio-data\") pod \"e2e-inf-b-953e38-openshift-default-5567878d9c-r76gn\" (UID: \"d9c803a7-01f3-409c-857f-48b2246e009f\") " pod="openshift-ingress/e2e-inf-b-953e38-openshift-default-5567878d9c-r76gn" Jul 13 22:43:59.013167 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:43:59.013083 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d9c803a7-01f3-409c-857f-48b2246e009f-istio-token\") pod \"e2e-inf-b-953e38-openshift-default-5567878d9c-r76gn\" (UID: \"d9c803a7-01f3-409c-857f-48b2246e009f\") " pod="openshift-ingress/e2e-inf-b-953e38-openshift-default-5567878d9c-r76gn" Jul 13 22:43:59.013167 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:43:59.013158 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d9c803a7-01f3-409c-857f-48b2246e009f-workload-certs\") pod \"e2e-inf-b-953e38-openshift-default-5567878d9c-r76gn\" (UID: \"d9c803a7-01f3-409c-857f-48b2246e009f\") " pod="openshift-ingress/e2e-inf-b-953e38-openshift-default-5567878d9c-r76gn" Jul 13 22:43:59.013371 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:43:59.013214 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d9c803a7-01f3-409c-857f-48b2246e009f-workload-socket\") pod \"e2e-inf-b-953e38-openshift-default-5567878d9c-r76gn\" (UID: \"d9c803a7-01f3-409c-857f-48b2246e009f\") " pod="openshift-ingress/e2e-inf-b-953e38-openshift-default-5567878d9c-r76gn" Jul 13 22:43:59.013371 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:43:59.013241 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d9c803a7-01f3-409c-857f-48b2246e009f-istio-podinfo\") pod \"e2e-inf-b-953e38-openshift-default-5567878d9c-r76gn\" (UID: \"d9c803a7-01f3-409c-857f-48b2246e009f\") " pod="openshift-ingress/e2e-inf-b-953e38-openshift-default-5567878d9c-r76gn" Jul 13 22:43:59.013371 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:43:59.013270 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2jfvp\" (UniqueName: \"kubernetes.io/projected/d9c803a7-01f3-409c-857f-48b2246e009f-kube-api-access-2jfvp\") pod \"e2e-inf-b-953e38-openshift-default-5567878d9c-r76gn\" (UID: \"d9c803a7-01f3-409c-857f-48b2246e009f\") " pod="openshift-ingress/e2e-inf-b-953e38-openshift-default-5567878d9c-r76gn" Jul 13 22:43:59.013525 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:43:59.013478 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d9c803a7-01f3-409c-857f-48b2246e009f-credential-socket\") pod \"e2e-inf-b-953e38-openshift-default-5567878d9c-r76gn\" (UID: \"d9c803a7-01f3-409c-857f-48b2246e009f\") " pod="openshift-ingress/e2e-inf-b-953e38-openshift-default-5567878d9c-r76gn" Jul 13 22:43:59.013754 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:43:59.013732 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d9c803a7-01f3-409c-857f-48b2246e009f-istiod-ca-cert\") pod \"e2e-inf-b-953e38-openshift-default-5567878d9c-r76gn\" (UID: \"d9c803a7-01f3-409c-857f-48b2246e009f\") " pod="openshift-ingress/e2e-inf-b-953e38-openshift-default-5567878d9c-r76gn" Jul 13 22:43:59.013823 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:43:59.013757 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d9c803a7-01f3-409c-857f-48b2246e009f-workload-socket\") pod \"e2e-inf-b-953e38-openshift-default-5567878d9c-r76gn\" (UID: \"d9c803a7-01f3-409c-857f-48b2246e009f\") " pod="openshift-ingress/e2e-inf-b-953e38-openshift-default-5567878d9c-r76gn" Jul 13 22:43:59.013823 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:43:59.013802 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d9c803a7-01f3-409c-857f-48b2246e009f-workload-certs\") pod \"e2e-inf-b-953e38-openshift-default-5567878d9c-r76gn\" (UID: \"d9c803a7-01f3-409c-857f-48b2246e009f\") " pod="openshift-ingress/e2e-inf-b-953e38-openshift-default-5567878d9c-r76gn" Jul 13 22:43:59.014013 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:43:59.013991 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d9c803a7-01f3-409c-857f-48b2246e009f-istio-data\") pod \"e2e-inf-b-953e38-openshift-default-5567878d9c-r76gn\" (UID: \"d9c803a7-01f3-409c-857f-48b2246e009f\") " pod="openshift-ingress/e2e-inf-b-953e38-openshift-default-5567878d9c-r76gn" Jul 13 22:43:59.015421 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:43:59.015386 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d9c803a7-01f3-409c-857f-48b2246e009f-istio-envoy\") pod \"e2e-inf-b-953e38-openshift-default-5567878d9c-r76gn\" (UID: \"d9c803a7-01f3-409c-857f-48b2246e009f\") " pod="openshift-ingress/e2e-inf-b-953e38-openshift-default-5567878d9c-r76gn" Jul 13 22:43:59.015867 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:43:59.015850 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d9c803a7-01f3-409c-857f-48b2246e009f-istio-podinfo\") pod \"e2e-inf-b-953e38-openshift-default-5567878d9c-r76gn\" (UID: \"d9c803a7-01f3-409c-857f-48b2246e009f\") " pod="openshift-ingress/e2e-inf-b-953e38-openshift-default-5567878d9c-r76gn" Jul 13 22:43:59.021604 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:43:59.021582 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d9c803a7-01f3-409c-857f-48b2246e009f-istio-token\") pod \"e2e-inf-b-953e38-openshift-default-5567878d9c-r76gn\" (UID: \"d9c803a7-01f3-409c-857f-48b2246e009f\") " pod="openshift-ingress/e2e-inf-b-953e38-openshift-default-5567878d9c-r76gn" Jul 13 22:43:59.027690 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:43:59.027665 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2jfvp\" (UniqueName: \"kubernetes.io/projected/d9c803a7-01f3-409c-857f-48b2246e009f-kube-api-access-2jfvp\") pod \"e2e-inf-b-953e38-openshift-default-5567878d9c-r76gn\" (UID: \"d9c803a7-01f3-409c-857f-48b2246e009f\") " pod="openshift-ingress/e2e-inf-b-953e38-openshift-default-5567878d9c-r76gn" Jul 13 22:43:59.113888 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:43:59.113799 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-953e38-openshift-default-5567878d9c-r76gn" Jul 13 22:43:59.262192 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:43:59.262127 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-953e38-openshift-default-5567878d9c-r76gn"] Jul 13 22:43:59.264761 ip-10-0-133-179 kubenswrapper[2583]: W0713 22:43:59.264731 2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd9c803a7_01f3_409c_857f_48b2246e009f.slice/crio-1ed466a7c4c40bc859ba2019eae404cbc05bc9a52b5311dbceb0a2cd57154965 WatchSource:0}: Error finding container 1ed466a7c4c40bc859ba2019eae404cbc05bc9a52b5311dbceb0a2cd57154965: Status 404 returned error can't find the container with id 1ed466a7c4c40bc859ba2019eae404cbc05bc9a52b5311dbceb0a2cd57154965 Jul 13 22:43:59.266883 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:43:59.266851 2583 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 13 22:43:59.266957 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:43:59.266923 2583 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 13 22:43:59.266993 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:43:59.266954 2583 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 13 22:43:59.758943 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:43:59.758908 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-953e38-openshift-default-5567878d9c-r76gn" event={"ID":"d9c803a7-01f3-409c-857f-48b2246e009f","Type":"ContainerStarted","Data":"d98537e258409ec0ba624123662fc645088a09e3bf6207517e40d9f050e37816"} Jul 13 22:43:59.759163 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:43:59.758971 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-953e38-openshift-default-5567878d9c-r76gn" event={"ID":"d9c803a7-01f3-409c-857f-48b2246e009f","Type":"ContainerStarted","Data":"1ed466a7c4c40bc859ba2019eae404cbc05bc9a52b5311dbceb0a2cd57154965"} Jul 13 22:43:59.783011 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:43:59.782945 2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-b-953e38-openshift-default-5567878d9c-r76gn" podStartSLOduration=1.782929765 podStartE2EDuration="1.782929765s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-13 22:43:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 22:43:59.781212729 +0000 UTC m=+1835.029128079" watchObservedRunningTime="2026-07-13 22:43:59.782929765 +0000 UTC m=+1835.030845115" Jul 13 22:44:00.115000 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:44:00.114955 2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-953e38-openshift-default-5567878d9c-r76gn" Jul 13 22:44:00.119946 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:44:00.119918 2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-953e38-openshift-default-5567878d9c-r76gn" Jul 13 22:44:00.763214 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:44:00.763183 2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-953e38-openshift-default-5567878d9c-r76gn" Jul 13 22:44:00.764247 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:44:00.764226 2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-953e38-openshift-default-5567878d9c-r76gn" Jul 13 22:44:01.752992 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:44:01.752952 2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-6e7f3c-7b4fc9f994-cbddg" Jul 13 22:44:01.844652 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:44:01.844615 2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-953e38-7888884cf8-pzwfv"] Jul 13 22:44:01.849366 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:44:01.849337 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-953e38-7888884cf8-pzwfv" Jul 13 22:44:01.852155 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:44:01.852129 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-b-953e38-serving-cert\"" Jul 13 22:44:01.857279 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:44:01.857249 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-953e38-7888884cf8-pzwfv"] Jul 13 22:44:01.946103 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:44:01.946055 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/01f8036d-8b11-4077-885a-bb5534ae9f43-maas-api-tls\") pod \"maas-api-e2e-inf-b-953e38-7888884cf8-pzwfv\" (UID: \"01f8036d-8b11-4077-885a-bb5534ae9f43\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-953e38-7888884cf8-pzwfv" Jul 13 22:44:01.946380 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:44:01.946360 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jczns\" (UniqueName: \"kubernetes.io/projected/01f8036d-8b11-4077-885a-bb5534ae9f43-kube-api-access-jczns\") pod \"maas-api-e2e-inf-b-953e38-7888884cf8-pzwfv\" (UID: \"01f8036d-8b11-4077-885a-bb5534ae9f43\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-953e38-7888884cf8-pzwfv" Jul 13 22:44:01.946552 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:44:01.946536 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/01f8036d-8b11-4077-885a-bb5534ae9f43-openshift-service-ca\") pod \"maas-api-e2e-inf-b-953e38-7888884cf8-pzwfv\" (UID: \"01f8036d-8b11-4077-885a-bb5534ae9f43\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-953e38-7888884cf8-pzwfv" Jul 13 22:44:02.048016 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:44:02.047961 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/01f8036d-8b11-4077-885a-bb5534ae9f43-openshift-service-ca\") pod \"maas-api-e2e-inf-b-953e38-7888884cf8-pzwfv\" (UID: \"01f8036d-8b11-4077-885a-bb5534ae9f43\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-953e38-7888884cf8-pzwfv" Jul 13 22:44:02.048210 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:44:02.048097 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/01f8036d-8b11-4077-885a-bb5534ae9f43-maas-api-tls\") pod \"maas-api-e2e-inf-b-953e38-7888884cf8-pzwfv\" (UID: \"01f8036d-8b11-4077-885a-bb5534ae9f43\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-953e38-7888884cf8-pzwfv" Jul 13 22:44:02.048275 ip-10-0-133-179 kubenswrapper[2583]: E0713 22:44:02.048251 2583 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-inf-b-953e38-serving-cert: secret "maas-api-e2e-inf-b-953e38-serving-cert" not found Jul 13 22:44:02.048362 ip-10-0-133-179 kubenswrapper[2583]: E0713 22:44:02.048321 2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/01f8036d-8b11-4077-885a-bb5534ae9f43-maas-api-tls podName:01f8036d-8b11-4077-885a-bb5534ae9f43 nodeName:}" failed. No retries permitted until 2026-07-13 22:44:02.548296362 +0000 UTC m=+1837.796211696 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/01f8036d-8b11-4077-885a-bb5534ae9f43-maas-api-tls") pod "maas-api-e2e-inf-b-953e38-7888884cf8-pzwfv" (UID: "01f8036d-8b11-4077-885a-bb5534ae9f43") : secret "maas-api-e2e-inf-b-953e38-serving-cert" not found Jul 13 22:44:02.049441 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:44:02.049405 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/01f8036d-8b11-4077-885a-bb5534ae9f43-openshift-service-ca\") pod \"maas-api-e2e-inf-b-953e38-7888884cf8-pzwfv\" (UID: \"01f8036d-8b11-4077-885a-bb5534ae9f43\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-953e38-7888884cf8-pzwfv" Jul 13 22:44:02.049559 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:44:02.049498 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jczns\" (UniqueName: \"kubernetes.io/projected/01f8036d-8b11-4077-885a-bb5534ae9f43-kube-api-access-jczns\") pod \"maas-api-e2e-inf-b-953e38-7888884cf8-pzwfv\" (UID: \"01f8036d-8b11-4077-885a-bb5534ae9f43\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-953e38-7888884cf8-pzwfv" Jul 13 22:44:02.061607 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:44:02.061493 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jczns\" (UniqueName: \"kubernetes.io/projected/01f8036d-8b11-4077-885a-bb5534ae9f43-kube-api-access-jczns\") pod \"maas-api-e2e-inf-b-953e38-7888884cf8-pzwfv\" (UID: \"01f8036d-8b11-4077-885a-bb5534ae9f43\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-953e38-7888884cf8-pzwfv" Jul 13 22:44:02.555096 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:44:02.555057 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/01f8036d-8b11-4077-885a-bb5534ae9f43-maas-api-tls\") pod \"maas-api-e2e-inf-b-953e38-7888884cf8-pzwfv\" (UID: \"01f8036d-8b11-4077-885a-bb5534ae9f43\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-953e38-7888884cf8-pzwfv" Jul 13 22:44:02.557689 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:44:02.557660 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/01f8036d-8b11-4077-885a-bb5534ae9f43-maas-api-tls\") pod \"maas-api-e2e-inf-b-953e38-7888884cf8-pzwfv\" (UID: \"01f8036d-8b11-4077-885a-bb5534ae9f43\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-953e38-7888884cf8-pzwfv" Jul 13 22:44:02.766683 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:44:02.766641 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-953e38-7888884cf8-pzwfv" Jul 13 22:44:02.949536 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:44:02.949354 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-953e38-7888884cf8-pzwfv"] Jul 13 22:44:03.778437 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:44:03.778398 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-953e38-7888884cf8-pzwfv" event={"ID":"01f8036d-8b11-4077-885a-bb5534ae9f43","Type":"ContainerStarted","Data":"926878540b88643301f4548ad5ab7801cdc0f9eb7a3bc53d9e5684c066a87247"} Jul 13 22:44:03.778932 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:44:03.778454 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-953e38-7888884cf8-pzwfv" event={"ID":"01f8036d-8b11-4077-885a-bb5534ae9f43","Type":"ContainerStarted","Data":"08fd3f45a3ffde38daad1e86662e0d73f94fc2cfda7f339bad9eff85ae957865"} Jul 13 22:44:03.778932 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:44:03.778499 2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-953e38-7888884cf8-pzwfv" Jul 13 22:44:03.795964 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:44:03.795911 2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-953e38-7888884cf8-pzwfv" podStartSLOduration=2.499000967 podStartE2EDuration="2.795894011s" totalImagesPullingTime="296.893044ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-13 22:44:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 22:44:03.794536882 +0000 UTC m=+1839.042452234" watchObservedRunningTime="2026-07-13 22:44:03.795894011 +0000 UTC m=+1839.043809361" Jul 13 22:44:07.728089 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:44:07.728053 2583 patch_prober.go:28] interesting pod/e2e-inf-a-6e7f3c-openshift-default-5f488c48df-w2rvd container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.33:15021/healthz/ready\": dial tcp 10.133.0.33:15021: connect: connection refused" start-of-body= Jul 13 22:44:07.728491 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:44:07.728116 2583 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-a-6e7f3c-openshift-default-5f488c48df-w2rvd" podUID="7bafdacd-97b2-440e-ab73-a3591de9b949" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.33:15021/healthz/ready\": dial tcp 10.133.0.33:15021: connect: connection refused" Jul 13 22:44:07.797808 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:44:07.797775 2583 generic.go:356] "Generic (PLEG): container finished" podID="7bafdacd-97b2-440e-ab73-a3591de9b949" containerID="9bfb85ecfd065b23dd78f95d5591e76bfc49c3f6ab59556885a32181a9863f4f" exitCode=0 Jul 13 22:44:07.797808 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:44:07.797812 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-6e7f3c-openshift-default-5f488c48df-w2rvd" event={"ID":"7bafdacd-97b2-440e-ab73-a3591de9b949","Type":"ContainerDied","Data":"9bfb85ecfd065b23dd78f95d5591e76bfc49c3f6ab59556885a32181a9863f4f"} Jul 13 22:44:07.798208 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:44:07.798194 2583 scope.go:117] "RemoveContainer" containerID="9bfb85ecfd065b23dd78f95d5591e76bfc49c3f6ab59556885a32181a9863f4f" Jul 13 22:44:07.799248 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:44:07.799216 2583 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 13 22:44:07.799301 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:44:07.799284 2583 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 13 22:44:07.799339 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:44:07.799319 2583 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 13 22:44:08.803895 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:44:08.803859 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-6e7f3c-openshift-default-5f488c48df-w2rvd" event={"ID":"7bafdacd-97b2-440e-ab73-a3591de9b949","Type":"ContainerStarted","Data":"ed84f253af9f4bf6c45cd4406421ef9e36bb6abb5c10ef266adbca3a7889edad"} Jul 13 22:44:09.141471 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:44:09.141365 2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-6e7f3c-openshift-default-5f488c48df-w2rvd" Jul 13 22:44:09.141814 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:44:09.141783 2583 patch_prober.go:28] interesting pod/e2e-inf-a-6e7f3c-openshift-default-5f488c48df-w2rvd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.33:15021/healthz/ready\": dial tcp 10.133.0.33:15021: connect: connection refused" start-of-body= Jul 13 22:44:09.141925 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:44:09.141848 2583 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-6e7f3c-openshift-default-5f488c48df-w2rvd" podUID="7bafdacd-97b2-440e-ab73-a3591de9b949" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.33:15021/healthz/ready\": dial tcp 10.133.0.33:15021: connect: connection refused" Jul 13 22:44:09.789945 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:44:09.789911 2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-953e38-7888884cf8-pzwfv" Jul 13 22:44:10.142772 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:44:10.142218 2583 patch_prober.go:28] interesting pod/e2e-inf-a-6e7f3c-openshift-default-5f488c48df-w2rvd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.33:15021/healthz/ready\": dial tcp 10.133.0.33:15021: connect: connection refused" start-of-body= Jul 13 22:44:10.142772 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:44:10.142284 2583 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-6e7f3c-openshift-default-5f488c48df-w2rvd" podUID="7bafdacd-97b2-440e-ab73-a3591de9b949" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.33:15021/healthz/ready\": dial tcp 10.133.0.33:15021: connect: connection refused" Jul 13 22:44:11.141648 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:44:11.141607 2583 patch_prober.go:28] interesting pod/e2e-inf-a-6e7f3c-openshift-default-5f488c48df-w2rvd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.33:15021/healthz/ready\": dial tcp 10.133.0.33:15021: connect: connection refused" start-of-body= Jul 13 22:44:11.141965 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:44:11.141927 2583 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-6e7f3c-openshift-default-5f488c48df-w2rvd" podUID="7bafdacd-97b2-440e-ab73-a3591de9b949" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.33:15021/healthz/ready\": dial tcp 10.133.0.33:15021: connect: connection refused" Jul 13 22:44:12.142386 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:44:12.142342 2583 patch_prober.go:28] interesting pod/e2e-inf-a-6e7f3c-openshift-default-5f488c48df-w2rvd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.33:15021/healthz/ready\": dial tcp 10.133.0.33:15021: connect: connection refused" start-of-body= Jul 13 22:44:12.142999 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:44:12.142964 2583 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-6e7f3c-openshift-default-5f488c48df-w2rvd" podUID="7bafdacd-97b2-440e-ab73-a3591de9b949" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.33:15021/healthz/ready\": dial tcp 10.133.0.33:15021: connect: connection refused" Jul 13 22:44:13.146552 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:44:13.146516 2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-6e7f3c-openshift-default-5f488c48df-w2rvd" Jul 13 22:44:13.147086 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:44:13.147026 2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-6e7f3c-openshift-default-5f488c48df-w2rvd" Jul 13 22:44:13.148007 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:44:13.147978 2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-6e7f3c-openshift-default-5f488c48df-w2rvd" Jul 13 22:44:15.764231 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:44:15.764132 2583 patch_prober.go:28] interesting pod/e2e-inf-b-953e38-openshift-default-5567878d9c-r76gn container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.35:15021/healthz/ready\": dial tcp 10.133.0.35:15021: connect: connection refused" start-of-body= Jul 13 22:44:15.764231 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:44:15.764195 2583 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-b-953e38-openshift-default-5567878d9c-r76gn" podUID="d9c803a7-01f3-409c-857f-48b2246e009f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.35:15021/healthz/ready\": dial tcp 10.133.0.35:15021: connect: connection refused" Jul 13 22:44:15.837377 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:44:15.837346 2583 generic.go:356] "Generic (PLEG): container finished" podID="d9c803a7-01f3-409c-857f-48b2246e009f" containerID="d98537e258409ec0ba624123662fc645088a09e3bf6207517e40d9f050e37816" exitCode=0 Jul 13 22:44:15.837545 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:44:15.837418 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-953e38-openshift-default-5567878d9c-r76gn" event={"ID":"d9c803a7-01f3-409c-857f-48b2246e009f","Type":"ContainerDied","Data":"d98537e258409ec0ba624123662fc645088a09e3bf6207517e40d9f050e37816"} Jul 13 22:44:15.837841 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:44:15.837828 2583 scope.go:117] "RemoveContainer" containerID="d98537e258409ec0ba624123662fc645088a09e3bf6207517e40d9f050e37816" Jul 13 22:44:15.838797 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:44:15.838770 2583 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 13 22:44:15.838870 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:44:15.838856 2583 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 13 22:44:15.838906 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:44:15.838890 2583 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 13 22:44:16.846415 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:44:16.846370 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-953e38-openshift-default-5567878d9c-r76gn" event={"ID":"d9c803a7-01f3-409c-857f-48b2246e009f","Type":"ContainerStarted","Data":"6220c761ef6d6a8ea40e5cd28f8e12841177b01692c3bf2a3a5984d0c7fa1c27"} Jul 13 22:44:17.114208 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:44:17.114111 2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-953e38-openshift-default-5567878d9c-r76gn" Jul 13 22:44:17.114370 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:44:17.114339 2583 patch_prober.go:28] interesting pod/e2e-inf-b-953e38-openshift-default-5567878d9c-r76gn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.35:15021/healthz/ready\": dial tcp 10.133.0.35:15021: connect: connection refused" start-of-body= Jul 13 22:44:17.114419 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:44:17.114381 2583 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-953e38-openshift-default-5567878d9c-r76gn" podUID="d9c803a7-01f3-409c-857f-48b2246e009f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.35:15021/healthz/ready\": dial tcp 10.133.0.35:15021: connect: connection refused" Jul 13 22:44:18.114370 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:44:18.114333 2583 patch_prober.go:28] interesting pod/e2e-inf-b-953e38-openshift-default-5567878d9c-r76gn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.35:15021/healthz/ready\": dial tcp 10.133.0.35:15021: connect: connection refused" start-of-body= Jul 13 22:44:18.114808 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:44:18.114386 2583 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-953e38-openshift-default-5567878d9c-r76gn" podUID="d9c803a7-01f3-409c-857f-48b2246e009f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.35:15021/healthz/ready\": dial tcp 10.133.0.35:15021: connect: connection refused" Jul 13 22:44:19.114943 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:44:19.114906 2583 patch_prober.go:28] interesting pod/e2e-inf-b-953e38-openshift-default-5567878d9c-r76gn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.35:15021/healthz/ready\": dial tcp 10.133.0.35:15021: connect: connection refused" start-of-body= Jul 13 22:44:19.115349 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:44:19.114962 2583 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-953e38-openshift-default-5567878d9c-r76gn" podUID="d9c803a7-01f3-409c-857f-48b2246e009f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.35:15021/healthz/ready\": dial tcp 10.133.0.35:15021: connect: connection refused" Jul 13 22:44:20.114815 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:44:20.114780 2583 patch_prober.go:28] interesting pod/e2e-inf-b-953e38-openshift-default-5567878d9c-r76gn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.35:15021/healthz/ready\": dial tcp 10.133.0.35:15021: connect: connection refused" start-of-body= Jul 13 22:44:20.114988 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:44:20.114835 2583 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-953e38-openshift-default-5567878d9c-r76gn" podUID="d9c803a7-01f3-409c-857f-48b2246e009f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.35:15021/healthz/ready\": dial tcp 10.133.0.35:15021: connect: connection refused" Jul 13 22:44:21.118632 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:44:21.118596 2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-953e38-openshift-default-5567878d9c-r76gn" Jul 13 22:44:21.119003 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:44:21.118866 2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-953e38-openshift-default-5567878d9c-r76gn" Jul 13 22:44:21.119450 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:44:21.119431 2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-953e38-openshift-default-5567878d9c-r76gn" Jul 13 22:45:22.089348 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:45:22.089310 2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-6e7f3c-7b4fc9f994-cbddg"] Jul 13 22:45:22.089959 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:45:22.089621 2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-6e7f3c-7b4fc9f994-cbddg" podUID="f13fb69b-5c84-43b2-8606-0607ab353004" containerName="maas-api" containerID="cri-o://4b35918e5ebfae35362f24642a343bf7b111462e9e026444daa630bcb5565506" gracePeriod=30 Jul 13 22:45:22.532182 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:45:22.532153 2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-6e7f3c-7b4fc9f994-cbddg" Jul 13 22:45:22.592152 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:45:22.592118 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f13fb69b-5c84-43b2-8606-0607ab353004-openshift-service-ca\") pod \"f13fb69b-5c84-43b2-8606-0607ab353004\" (UID: \"f13fb69b-5c84-43b2-8606-0607ab353004\") " Jul 13 22:45:22.592352 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:45:22.592212 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-pscxq\" (UniqueName: \"kubernetes.io/projected/f13fb69b-5c84-43b2-8606-0607ab353004-kube-api-access-pscxq\") pod \"f13fb69b-5c84-43b2-8606-0607ab353004\" (UID: \"f13fb69b-5c84-43b2-8606-0607ab353004\") " Jul 13 22:45:22.592352 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:45:22.592324 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f13fb69b-5c84-43b2-8606-0607ab353004-maas-api-tls\") pod \"f13fb69b-5c84-43b2-8606-0607ab353004\" (UID: \"f13fb69b-5c84-43b2-8606-0607ab353004\") " Jul 13 22:45:22.592519 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:45:22.592499 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f13fb69b-5c84-43b2-8606-0607ab353004-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "f13fb69b-5c84-43b2-8606-0607ab353004" (UID: "f13fb69b-5c84-43b2-8606-0607ab353004"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 13 22:45:22.592689 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:45:22.592669 2583 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f13fb69b-5c84-43b2-8606-0607ab353004-openshift-service-ca\") on node \"ip-10-0-133-179.ec2.internal\" DevicePath \"\"" Jul 13 22:45:22.594425 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:45:22.594396 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f13fb69b-5c84-43b2-8606-0607ab353004-kube-api-access-pscxq" (OuterVolumeSpecName: "kube-api-access-pscxq") pod "f13fb69b-5c84-43b2-8606-0607ab353004" (UID: "f13fb69b-5c84-43b2-8606-0607ab353004"). InnerVolumeSpecName "kube-api-access-pscxq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 13 22:45:22.594515 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:45:22.594451 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f13fb69b-5c84-43b2-8606-0607ab353004-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "f13fb69b-5c84-43b2-8606-0607ab353004" (UID: "f13fb69b-5c84-43b2-8606-0607ab353004"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 13 22:45:22.694032 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:45:22.693947 2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pscxq\" (UniqueName: \"kubernetes.io/projected/f13fb69b-5c84-43b2-8606-0607ab353004-kube-api-access-pscxq\") on node \"ip-10-0-133-179.ec2.internal\" DevicePath \"\"" Jul 13 22:45:22.694032 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:45:22.693976 2583 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f13fb69b-5c84-43b2-8606-0607ab353004-maas-api-tls\") on node \"ip-10-0-133-179.ec2.internal\" DevicePath \"\"" Jul 13 22:45:23.135637 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:45:23.135594 2583 generic.go:356] "Generic (PLEG): container finished" podID="f13fb69b-5c84-43b2-8606-0607ab353004" containerID="4b35918e5ebfae35362f24642a343bf7b111462e9e026444daa630bcb5565506" exitCode=0 Jul 13 22:45:23.135637 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:45:23.135646 2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-6e7f3c-7b4fc9f994-cbddg" Jul 13 22:45:23.136191 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:45:23.135674 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-6e7f3c-7b4fc9f994-cbddg" event={"ID":"f13fb69b-5c84-43b2-8606-0607ab353004","Type":"ContainerDied","Data":"4b35918e5ebfae35362f24642a343bf7b111462e9e026444daa630bcb5565506"} Jul 13 22:45:23.136191 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:45:23.135731 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-6e7f3c-7b4fc9f994-cbddg" event={"ID":"f13fb69b-5c84-43b2-8606-0607ab353004","Type":"ContainerDied","Data":"38a9d66548624d265ab27f960881d9de5c03df6dac8f0670284d401a9b763ac9"} Jul 13 22:45:23.136191 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:45:23.135754 2583 scope.go:117] "RemoveContainer" containerID="4b35918e5ebfae35362f24642a343bf7b111462e9e026444daa630bcb5565506" Jul 13 22:45:23.144419 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:45:23.144404 2583 scope.go:117] "RemoveContainer" containerID="4b35918e5ebfae35362f24642a343bf7b111462e9e026444daa630bcb5565506" Jul 13 22:45:23.144703 ip-10-0-133-179 kubenswrapper[2583]: E0713 22:45:23.144682 2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4b35918e5ebfae35362f24642a343bf7b111462e9e026444daa630bcb5565506\": container with ID starting with 4b35918e5ebfae35362f24642a343bf7b111462e9e026444daa630bcb5565506 not found: ID does not exist" containerID="4b35918e5ebfae35362f24642a343bf7b111462e9e026444daa630bcb5565506" Jul 13 22:45:23.144778 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:45:23.144716 2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4b35918e5ebfae35362f24642a343bf7b111462e9e026444daa630bcb5565506"} err="failed to get container status \"4b35918e5ebfae35362f24642a343bf7b111462e9e026444daa630bcb5565506\": rpc error: code = NotFound desc = could not find container \"4b35918e5ebfae35362f24642a343bf7b111462e9e026444daa630bcb5565506\": container with ID starting with 4b35918e5ebfae35362f24642a343bf7b111462e9e026444daa630bcb5565506 not found: ID does not exist" Jul 13 22:45:23.155928 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:45:23.155881 2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-6e7f3c-7b4fc9f994-cbddg"] Jul 13 22:45:23.157834 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:45:23.157811 2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-6e7f3c-7b4fc9f994-cbddg"] Jul 13 22:45:23.390060 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:45:23.389978 2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f13fb69b-5c84-43b2-8606-0607ab353004" path="/var/lib/kubelet/pods/f13fb69b-5c84-43b2-8606-0607ab353004/volumes" Jul 13 22:46:03.817667 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:46:03.817625 2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-6e7f3c-openshift-default-5f488c48df-w2rvd"] Jul 13 22:46:03.818295 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:46:03.817869 2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-6e7f3c-openshift-default-5f488c48df-w2rvd" podUID="7bafdacd-97b2-440e-ab73-a3591de9b949" containerName="istio-proxy" containerID="cri-o://ed84f253af9f4bf6c45cd4406421ef9e36bb6abb5c10ef266adbca3a7889edad" gracePeriod=30 Jul 13 22:46:05.016587 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:46:05.016540 2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-953e38-7888884cf8-pzwfv"] Jul 13 22:46:05.017002 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:46:05.016783 2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-953e38-7888884cf8-pzwfv" podUID="01f8036d-8b11-4077-885a-bb5534ae9f43" containerName="maas-api" containerID="cri-o://926878540b88643301f4548ad5ab7801cdc0f9eb7a3bc53d9e5684c066a87247" gracePeriod=30 Jul 13 22:46:05.266993 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:46:05.266934 2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-953e38-7888884cf8-pzwfv" Jul 13 22:46:05.308136 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:46:05.308100 2583 generic.go:356] "Generic (PLEG): container finished" podID="01f8036d-8b11-4077-885a-bb5534ae9f43" containerID="926878540b88643301f4548ad5ab7801cdc0f9eb7a3bc53d9e5684c066a87247" exitCode=0 Jul 13 22:46:05.308317 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:46:05.308153 2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-953e38-7888884cf8-pzwfv" Jul 13 22:46:05.308317 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:46:05.308169 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-953e38-7888884cf8-pzwfv" event={"ID":"01f8036d-8b11-4077-885a-bb5534ae9f43","Type":"ContainerDied","Data":"926878540b88643301f4548ad5ab7801cdc0f9eb7a3bc53d9e5684c066a87247"} Jul 13 22:46:05.308317 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:46:05.308237 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-953e38-7888884cf8-pzwfv" event={"ID":"01f8036d-8b11-4077-885a-bb5534ae9f43","Type":"ContainerDied","Data":"08fd3f45a3ffde38daad1e86662e0d73f94fc2cfda7f339bad9eff85ae957865"} Jul 13 22:46:05.308317 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:46:05.308260 2583 scope.go:117] "RemoveContainer" containerID="926878540b88643301f4548ad5ab7801cdc0f9eb7a3bc53d9e5684c066a87247" Jul 13 22:46:05.316372 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:46:05.316354 2583 scope.go:117] "RemoveContainer" containerID="926878540b88643301f4548ad5ab7801cdc0f9eb7a3bc53d9e5684c066a87247" Jul 13 22:46:05.316680 ip-10-0-133-179 kubenswrapper[2583]: E0713 22:46:05.316659 2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"926878540b88643301f4548ad5ab7801cdc0f9eb7a3bc53d9e5684c066a87247\": container with ID starting with 926878540b88643301f4548ad5ab7801cdc0f9eb7a3bc53d9e5684c066a87247 not found: ID does not exist" containerID="926878540b88643301f4548ad5ab7801cdc0f9eb7a3bc53d9e5684c066a87247" Jul 13 22:46:05.316744 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:46:05.316690 2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"926878540b88643301f4548ad5ab7801cdc0f9eb7a3bc53d9e5684c066a87247"} err="failed to get container status \"926878540b88643301f4548ad5ab7801cdc0f9eb7a3bc53d9e5684c066a87247\": rpc error: code = NotFound desc = could not find container \"926878540b88643301f4548ad5ab7801cdc0f9eb7a3bc53d9e5684c066a87247\": container with ID starting with 926878540b88643301f4548ad5ab7801cdc0f9eb7a3bc53d9e5684c066a87247 not found: ID does not exist" Jul 13 22:46:05.374281 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:46:05.374244 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jczns\" (UniqueName: \"kubernetes.io/projected/01f8036d-8b11-4077-885a-bb5534ae9f43-kube-api-access-jczns\") pod \"01f8036d-8b11-4077-885a-bb5534ae9f43\" (UID: \"01f8036d-8b11-4077-885a-bb5534ae9f43\") " Jul 13 22:46:05.374474 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:46:05.374301 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/01f8036d-8b11-4077-885a-bb5534ae9f43-maas-api-tls\") pod \"01f8036d-8b11-4077-885a-bb5534ae9f43\" (UID: \"01f8036d-8b11-4077-885a-bb5534ae9f43\") " Jul 13 22:46:05.374474 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:46:05.374358 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/01f8036d-8b11-4077-885a-bb5534ae9f43-openshift-service-ca\") pod \"01f8036d-8b11-4077-885a-bb5534ae9f43\" (UID: \"01f8036d-8b11-4077-885a-bb5534ae9f43\") " Jul 13 22:46:05.374785 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:46:05.374761 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/01f8036d-8b11-4077-885a-bb5534ae9f43-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "01f8036d-8b11-4077-885a-bb5534ae9f43" (UID: "01f8036d-8b11-4077-885a-bb5534ae9f43"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 13 22:46:05.376434 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:46:05.376413 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/01f8036d-8b11-4077-885a-bb5534ae9f43-kube-api-access-jczns" (OuterVolumeSpecName: "kube-api-access-jczns") pod "01f8036d-8b11-4077-885a-bb5534ae9f43" (UID: "01f8036d-8b11-4077-885a-bb5534ae9f43"). InnerVolumeSpecName "kube-api-access-jczns". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 13 22:46:05.376434 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:46:05.376421 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/01f8036d-8b11-4077-885a-bb5534ae9f43-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "01f8036d-8b11-4077-885a-bb5534ae9f43" (UID: "01f8036d-8b11-4077-885a-bb5534ae9f43"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 13 22:46:05.475049 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:46:05.475013 2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jczns\" (UniqueName: \"kubernetes.io/projected/01f8036d-8b11-4077-885a-bb5534ae9f43-kube-api-access-jczns\") on node \"ip-10-0-133-179.ec2.internal\" DevicePath \"\"" Jul 13 22:46:05.475049 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:46:05.475046 2583 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/01f8036d-8b11-4077-885a-bb5534ae9f43-maas-api-tls\") on node \"ip-10-0-133-179.ec2.internal\" DevicePath \"\"" Jul 13 22:46:05.475049 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:46:05.475055 2583 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/01f8036d-8b11-4077-885a-bb5534ae9f43-openshift-service-ca\") on node \"ip-10-0-133-179.ec2.internal\" DevicePath \"\"" Jul 13 22:46:05.624028 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:46:05.623981 2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-953e38-7888884cf8-pzwfv"] Jul 13 22:46:05.626469 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:46:05.626436 2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-953e38-7888884cf8-pzwfv"] Jul 13 22:46:07.389653 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:46:07.389613 2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="01f8036d-8b11-4077-885a-bb5534ae9f43" path="/var/lib/kubelet/pods/01f8036d-8b11-4077-885a-bb5534ae9f43/volumes" Jul 13 22:46:09.070389 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:46:09.070366 2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-6e7f3c-openshift-default-5f488c48df-w2rvd" Jul 13 22:46:09.105857 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:46:09.105779 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7bafdacd-97b2-440e-ab73-a3591de9b949-istio-token\") pod \"7bafdacd-97b2-440e-ab73-a3591de9b949\" (UID: \"7bafdacd-97b2-440e-ab73-a3591de9b949\") " Jul 13 22:46:09.105857 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:46:09.105824 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7bafdacd-97b2-440e-ab73-a3591de9b949-istiod-ca-cert\") pod \"7bafdacd-97b2-440e-ab73-a3591de9b949\" (UID: \"7bafdacd-97b2-440e-ab73-a3591de9b949\") " Jul 13 22:46:09.106044 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:46:09.105942 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7bafdacd-97b2-440e-ab73-a3591de9b949-istio-podinfo\") pod \"7bafdacd-97b2-440e-ab73-a3591de9b949\" (UID: \"7bafdacd-97b2-440e-ab73-a3591de9b949\") " Jul 13 22:46:09.106044 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:46:09.105971 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7bafdacd-97b2-440e-ab73-a3591de9b949-credential-socket\") pod \"7bafdacd-97b2-440e-ab73-a3591de9b949\" (UID: \"7bafdacd-97b2-440e-ab73-a3591de9b949\") " Jul 13 22:46:09.106044 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:46:09.106001 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7bafdacd-97b2-440e-ab73-a3591de9b949-workload-certs\") pod \"7bafdacd-97b2-440e-ab73-a3591de9b949\" (UID: \"7bafdacd-97b2-440e-ab73-a3591de9b949\") " Jul 13 22:46:09.106044 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:46:09.106026 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7bafdacd-97b2-440e-ab73-a3591de9b949-workload-socket\") pod \"7bafdacd-97b2-440e-ab73-a3591de9b949\" (UID: \"7bafdacd-97b2-440e-ab73-a3591de9b949\") " Jul 13 22:46:09.106260 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:46:09.106054 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7bafdacd-97b2-440e-ab73-a3591de9b949-istio-envoy\") pod \"7bafdacd-97b2-440e-ab73-a3591de9b949\" (UID: \"7bafdacd-97b2-440e-ab73-a3591de9b949\") " Jul 13 22:46:09.106260 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:46:09.106095 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7bafdacd-97b2-440e-ab73-a3591de9b949-istio-data\") pod \"7bafdacd-97b2-440e-ab73-a3591de9b949\" (UID: \"7bafdacd-97b2-440e-ab73-a3591de9b949\") " Jul 13 22:46:09.106260 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:46:09.106126 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-dkj92\" (UniqueName: \"kubernetes.io/projected/7bafdacd-97b2-440e-ab73-a3591de9b949-kube-api-access-dkj92\") pod \"7bafdacd-97b2-440e-ab73-a3591de9b949\" (UID: \"7bafdacd-97b2-440e-ab73-a3591de9b949\") " Jul 13 22:46:09.106260 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:46:09.106238 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7bafdacd-97b2-440e-ab73-a3591de9b949-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "7bafdacd-97b2-440e-ab73-a3591de9b949" (UID: "7bafdacd-97b2-440e-ab73-a3591de9b949"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 13 22:46:09.106468 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:46:09.106263 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7bafdacd-97b2-440e-ab73-a3591de9b949-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "7bafdacd-97b2-440e-ab73-a3591de9b949" (UID: "7bafdacd-97b2-440e-ab73-a3591de9b949"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 13 22:46:09.106526 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:46:09.106471 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7bafdacd-97b2-440e-ab73-a3591de9b949-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "7bafdacd-97b2-440e-ab73-a3591de9b949" (UID: "7bafdacd-97b2-440e-ab73-a3591de9b949"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 13 22:46:09.106526 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:46:09.106503 2583 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7bafdacd-97b2-440e-ab73-a3591de9b949-credential-socket\") on node \"ip-10-0-133-179.ec2.internal\" DevicePath \"\"" Jul 13 22:46:09.106526 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:46:09.106522 2583 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7bafdacd-97b2-440e-ab73-a3591de9b949-istiod-ca-cert\") on node \"ip-10-0-133-179.ec2.internal\" DevicePath \"\"" Jul 13 22:46:09.106838 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:46:09.106804 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7bafdacd-97b2-440e-ab73-a3591de9b949-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "7bafdacd-97b2-440e-ab73-a3591de9b949" (UID: "7bafdacd-97b2-440e-ab73-a3591de9b949"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 13 22:46:09.107216 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:46:09.107182 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7bafdacd-97b2-440e-ab73-a3591de9b949-istio-data" (OuterVolumeSpecName: "istio-data") pod "7bafdacd-97b2-440e-ab73-a3591de9b949" (UID: "7bafdacd-97b2-440e-ab73-a3591de9b949"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 13 22:46:09.108490 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:46:09.108464 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7bafdacd-97b2-440e-ab73-a3591de9b949-kube-api-access-dkj92" (OuterVolumeSpecName: "kube-api-access-dkj92") pod "7bafdacd-97b2-440e-ab73-a3591de9b949" (UID: "7bafdacd-97b2-440e-ab73-a3591de9b949"). InnerVolumeSpecName "kube-api-access-dkj92". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 13 22:46:09.108724 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:46:09.108701 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7bafdacd-97b2-440e-ab73-a3591de9b949-istio-token" (OuterVolumeSpecName: "istio-token") pod "7bafdacd-97b2-440e-ab73-a3591de9b949" (UID: "7bafdacd-97b2-440e-ab73-a3591de9b949"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 13 22:46:09.108807 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:46:09.108698 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/7bafdacd-97b2-440e-ab73-a3591de9b949-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "7bafdacd-97b2-440e-ab73-a3591de9b949" (UID: "7bafdacd-97b2-440e-ab73-a3591de9b949"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 13 22:46:09.108807 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:46:09.108726 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7bafdacd-97b2-440e-ab73-a3591de9b949-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "7bafdacd-97b2-440e-ab73-a3591de9b949" (UID: "7bafdacd-97b2-440e-ab73-a3591de9b949"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 13 22:46:09.207359 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:46:09.207320 2583 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7bafdacd-97b2-440e-ab73-a3591de9b949-istio-token\") on node \"ip-10-0-133-179.ec2.internal\" DevicePath \"\"" Jul 13 22:46:09.207359 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:46:09.207352 2583 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7bafdacd-97b2-440e-ab73-a3591de9b949-istio-podinfo\") on node \"ip-10-0-133-179.ec2.internal\" DevicePath \"\"" Jul 13 22:46:09.207359 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:46:09.207364 2583 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7bafdacd-97b2-440e-ab73-a3591de9b949-workload-certs\") on node \"ip-10-0-133-179.ec2.internal\" DevicePath \"\"" Jul 13 22:46:09.207647 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:46:09.207375 2583 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7bafdacd-97b2-440e-ab73-a3591de9b949-workload-socket\") on node \"ip-10-0-133-179.ec2.internal\" DevicePath \"\"" Jul 13 22:46:09.207647 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:46:09.207387 2583 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7bafdacd-97b2-440e-ab73-a3591de9b949-istio-envoy\") on node \"ip-10-0-133-179.ec2.internal\" DevicePath \"\"" Jul 13 22:46:09.207647 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:46:09.207397 2583 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7bafdacd-97b2-440e-ab73-a3591de9b949-istio-data\") on node \"ip-10-0-133-179.ec2.internal\" DevicePath \"\"" Jul 13 22:46:09.207647 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:46:09.207408 2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dkj92\" (UniqueName: \"kubernetes.io/projected/7bafdacd-97b2-440e-ab73-a3591de9b949-kube-api-access-dkj92\") on node \"ip-10-0-133-179.ec2.internal\" DevicePath \"\"" Jul 13 22:46:09.327152 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:46:09.327114 2583 generic.go:356] "Generic (PLEG): container finished" podID="7bafdacd-97b2-440e-ab73-a3591de9b949" containerID="ed84f253af9f4bf6c45cd4406421ef9e36bb6abb5c10ef266adbca3a7889edad" exitCode=0 Jul 13 22:46:09.327320 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:46:09.327164 2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-6e7f3c-openshift-default-5f488c48df-w2rvd" Jul 13 22:46:09.327320 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:46:09.327177 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-6e7f3c-openshift-default-5f488c48df-w2rvd" event={"ID":"7bafdacd-97b2-440e-ab73-a3591de9b949","Type":"ContainerDied","Data":"ed84f253af9f4bf6c45cd4406421ef9e36bb6abb5c10ef266adbca3a7889edad"} Jul 13 22:46:09.327320 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:46:09.327220 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-6e7f3c-openshift-default-5f488c48df-w2rvd" event={"ID":"7bafdacd-97b2-440e-ab73-a3591de9b949","Type":"ContainerDied","Data":"ddd22d904cd1ef9871fd7ebb053668d13c054486b867c0ed462e381c476d8d1b"} Jul 13 22:46:09.327320 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:46:09.327242 2583 scope.go:117] "RemoveContainer" containerID="ed84f253af9f4bf6c45cd4406421ef9e36bb6abb5c10ef266adbca3a7889edad" Jul 13 22:46:09.335823 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:46:09.335805 2583 scope.go:117] "RemoveContainer" containerID="9bfb85ecfd065b23dd78f95d5591e76bfc49c3f6ab59556885a32181a9863f4f" Jul 13 22:46:09.343102 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:46:09.343087 2583 scope.go:117] "RemoveContainer" containerID="ed84f253af9f4bf6c45cd4406421ef9e36bb6abb5c10ef266adbca3a7889edad" Jul 13 22:46:09.343362 ip-10-0-133-179 kubenswrapper[2583]: E0713 22:46:09.343340 2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ed84f253af9f4bf6c45cd4406421ef9e36bb6abb5c10ef266adbca3a7889edad\": container with ID starting with ed84f253af9f4bf6c45cd4406421ef9e36bb6abb5c10ef266adbca3a7889edad not found: ID does not exist" containerID="ed84f253af9f4bf6c45cd4406421ef9e36bb6abb5c10ef266adbca3a7889edad" Jul 13 22:46:09.343426 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:46:09.343369 2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ed84f253af9f4bf6c45cd4406421ef9e36bb6abb5c10ef266adbca3a7889edad"} err="failed to get container status \"ed84f253af9f4bf6c45cd4406421ef9e36bb6abb5c10ef266adbca3a7889edad\": rpc error: code = NotFound desc = could not find container \"ed84f253af9f4bf6c45cd4406421ef9e36bb6abb5c10ef266adbca3a7889edad\": container with ID starting with ed84f253af9f4bf6c45cd4406421ef9e36bb6abb5c10ef266adbca3a7889edad not found: ID does not exist" Jul 13 22:46:09.343426 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:46:09.343387 2583 scope.go:117] "RemoveContainer" containerID="9bfb85ecfd065b23dd78f95d5591e76bfc49c3f6ab59556885a32181a9863f4f" Jul 13 22:46:09.343685 ip-10-0-133-179 kubenswrapper[2583]: E0713 22:46:09.343667 2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9bfb85ecfd065b23dd78f95d5591e76bfc49c3f6ab59556885a32181a9863f4f\": container with ID starting with 9bfb85ecfd065b23dd78f95d5591e76bfc49c3f6ab59556885a32181a9863f4f not found: ID does not exist" containerID="9bfb85ecfd065b23dd78f95d5591e76bfc49c3f6ab59556885a32181a9863f4f" Jul 13 22:46:09.343731 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:46:09.343691 2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9bfb85ecfd065b23dd78f95d5591e76bfc49c3f6ab59556885a32181a9863f4f"} err="failed to get container status \"9bfb85ecfd065b23dd78f95d5591e76bfc49c3f6ab59556885a32181a9863f4f\": rpc error: code = NotFound desc = could not find container \"9bfb85ecfd065b23dd78f95d5591e76bfc49c3f6ab59556885a32181a9863f4f\": container with ID starting with 9bfb85ecfd065b23dd78f95d5591e76bfc49c3f6ab59556885a32181a9863f4f not found: ID does not exist" Jul 13 22:46:09.347916 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:46:09.347891 2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-6e7f3c-openshift-default-5f488c48df-w2rvd"] Jul 13 22:46:09.354895 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:46:09.354876 2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-6e7f3c-openshift-default-5f488c48df-w2rvd"] Jul 13 22:46:09.389580 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:46:09.389498 2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7bafdacd-97b2-440e-ab73-a3591de9b949" path="/var/lib/kubelet/pods/7bafdacd-97b2-440e-ab73-a3591de9b949/volumes" Jul 13 22:46:10.828236 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:46:10.828198 2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-953e38-openshift-default-5567878d9c-r76gn"] Jul 13 22:46:10.828734 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:46:10.828471 2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-b-953e38-openshift-default-5567878d9c-r76gn" podUID="d9c803a7-01f3-409c-857f-48b2246e009f" containerName="istio-proxy" containerID="cri-o://6220c761ef6d6a8ea40e5cd28f8e12841177b01692c3bf2a3a5984d0c7fa1c27" gracePeriod=30 Jul 13 22:46:11.710531 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:46:11.710491 2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-a-8f95ee-openshift-default-5df565bcf9-whs2f"] Jul 13 22:46:11.710825 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:46:11.710809 2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="01f8036d-8b11-4077-885a-bb5534ae9f43" containerName="maas-api" Jul 13 22:46:11.710894 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:46:11.710827 2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="01f8036d-8b11-4077-885a-bb5534ae9f43" containerName="maas-api" Jul 13 22:46:11.710894 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:46:11.710838 2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f13fb69b-5c84-43b2-8606-0607ab353004" containerName="maas-api" Jul 13 22:46:11.710894 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:46:11.710843 2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="f13fb69b-5c84-43b2-8606-0607ab353004" containerName="maas-api" Jul 13 22:46:11.710894 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:46:11.710850 2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7bafdacd-97b2-440e-ab73-a3591de9b949" containerName="istio-proxy" Jul 13 22:46:11.710894 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:46:11.710856 2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="7bafdacd-97b2-440e-ab73-a3591de9b949" containerName="istio-proxy" Jul 13 22:46:11.710894 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:46:11.710863 2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7bafdacd-97b2-440e-ab73-a3591de9b949" containerName="istio-proxy" Jul 13 22:46:11.710894 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:46:11.710868 2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="7bafdacd-97b2-440e-ab73-a3591de9b949" containerName="istio-proxy" Jul 13 22:46:11.711260 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:46:11.710926 2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="01f8036d-8b11-4077-885a-bb5534ae9f43" containerName="maas-api" Jul 13 22:46:11.711260 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:46:11.710933 2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="7bafdacd-97b2-440e-ab73-a3591de9b949" containerName="istio-proxy" Jul 13 22:46:11.711260 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:46:11.710939 2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="7bafdacd-97b2-440e-ab73-a3591de9b949" containerName="istio-proxy" Jul 13 22:46:11.711260 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:46:11.710946 2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="f13fb69b-5c84-43b2-8606-0607ab353004" containerName="maas-api" Jul 13 22:46:11.715682 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:46:11.715658 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-8f95ee-openshift-default-5df565bcf9-whs2f" Jul 13 22:46:11.717844 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:46:11.717823 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-a-8f95ee-openshift-default-dockercfg-sxgck\"" Jul 13 22:46:11.725926 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:46:11.725905 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-8f95ee-openshift-default-5df565bcf9-whs2f"] Jul 13 22:46:11.826226 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:46:11.826186 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/950c3e6e-75a8-4e20-ba7f-84c0a08e1c1b-workload-socket\") pod \"e2e-api-a-8f95ee-openshift-default-5df565bcf9-whs2f\" (UID: \"950c3e6e-75a8-4e20-ba7f-84c0a08e1c1b\") " pod="openshift-ingress/e2e-api-a-8f95ee-openshift-default-5df565bcf9-whs2f" Jul 13 22:46:11.826226 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:46:11.826221 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/950c3e6e-75a8-4e20-ba7f-84c0a08e1c1b-istio-podinfo\") pod \"e2e-api-a-8f95ee-openshift-default-5df565bcf9-whs2f\" (UID: \"950c3e6e-75a8-4e20-ba7f-84c0a08e1c1b\") " pod="openshift-ingress/e2e-api-a-8f95ee-openshift-default-5df565bcf9-whs2f" Jul 13 22:46:11.826453 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:46:11.826250 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/950c3e6e-75a8-4e20-ba7f-84c0a08e1c1b-credential-socket\") pod \"e2e-api-a-8f95ee-openshift-default-5df565bcf9-whs2f\" (UID: \"950c3e6e-75a8-4e20-ba7f-84c0a08e1c1b\") " pod="openshift-ingress/e2e-api-a-8f95ee-openshift-default-5df565bcf9-whs2f" Jul 13 22:46:11.826453 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:46:11.826284 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/950c3e6e-75a8-4e20-ba7f-84c0a08e1c1b-istio-data\") pod \"e2e-api-a-8f95ee-openshift-default-5df565bcf9-whs2f\" (UID: \"950c3e6e-75a8-4e20-ba7f-84c0a08e1c1b\") " pod="openshift-ingress/e2e-api-a-8f95ee-openshift-default-5df565bcf9-whs2f" Jul 13 22:46:11.826453 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:46:11.826303 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/950c3e6e-75a8-4e20-ba7f-84c0a08e1c1b-istiod-ca-cert\") pod \"e2e-api-a-8f95ee-openshift-default-5df565bcf9-whs2f\" (UID: \"950c3e6e-75a8-4e20-ba7f-84c0a08e1c1b\") " pod="openshift-ingress/e2e-api-a-8f95ee-openshift-default-5df565bcf9-whs2f" Jul 13 22:46:11.826453 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:46:11.826337 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/950c3e6e-75a8-4e20-ba7f-84c0a08e1c1b-istio-envoy\") pod \"e2e-api-a-8f95ee-openshift-default-5df565bcf9-whs2f\" (UID: \"950c3e6e-75a8-4e20-ba7f-84c0a08e1c1b\") " pod="openshift-ingress/e2e-api-a-8f95ee-openshift-default-5df565bcf9-whs2f" Jul 13 22:46:11.826453 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:46:11.826403 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2f854\" (UniqueName: \"kubernetes.io/projected/950c3e6e-75a8-4e20-ba7f-84c0a08e1c1b-kube-api-access-2f854\") pod \"e2e-api-a-8f95ee-openshift-default-5df565bcf9-whs2f\" (UID: \"950c3e6e-75a8-4e20-ba7f-84c0a08e1c1b\") " pod="openshift-ingress/e2e-api-a-8f95ee-openshift-default-5df565bcf9-whs2f" Jul 13 22:46:11.826664 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:46:11.826454 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/950c3e6e-75a8-4e20-ba7f-84c0a08e1c1b-istio-token\") pod \"e2e-api-a-8f95ee-openshift-default-5df565bcf9-whs2f\" (UID: \"950c3e6e-75a8-4e20-ba7f-84c0a08e1c1b\") " pod="openshift-ingress/e2e-api-a-8f95ee-openshift-default-5df565bcf9-whs2f" Jul 13 22:46:11.826664 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:46:11.826534 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/950c3e6e-75a8-4e20-ba7f-84c0a08e1c1b-workload-certs\") pod \"e2e-api-a-8f95ee-openshift-default-5df565bcf9-whs2f\" (UID: \"950c3e6e-75a8-4e20-ba7f-84c0a08e1c1b\") " pod="openshift-ingress/e2e-api-a-8f95ee-openshift-default-5df565bcf9-whs2f" Jul 13 22:46:11.926992 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:46:11.926956 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/950c3e6e-75a8-4e20-ba7f-84c0a08e1c1b-workload-certs\") pod \"e2e-api-a-8f95ee-openshift-default-5df565bcf9-whs2f\" (UID: \"950c3e6e-75a8-4e20-ba7f-84c0a08e1c1b\") " pod="openshift-ingress/e2e-api-a-8f95ee-openshift-default-5df565bcf9-whs2f" Jul 13 22:46:11.927471 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:46:11.927001 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/950c3e6e-75a8-4e20-ba7f-84c0a08e1c1b-workload-socket\") pod \"e2e-api-a-8f95ee-openshift-default-5df565bcf9-whs2f\" (UID: \"950c3e6e-75a8-4e20-ba7f-84c0a08e1c1b\") " pod="openshift-ingress/e2e-api-a-8f95ee-openshift-default-5df565bcf9-whs2f" Jul 13 22:46:11.927471 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:46:11.927021 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/950c3e6e-75a8-4e20-ba7f-84c0a08e1c1b-istio-podinfo\") pod \"e2e-api-a-8f95ee-openshift-default-5df565bcf9-whs2f\" (UID: \"950c3e6e-75a8-4e20-ba7f-84c0a08e1c1b\") " pod="openshift-ingress/e2e-api-a-8f95ee-openshift-default-5df565bcf9-whs2f" Jul 13 22:46:11.927471 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:46:11.927121 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/950c3e6e-75a8-4e20-ba7f-84c0a08e1c1b-credential-socket\") pod \"e2e-api-a-8f95ee-openshift-default-5df565bcf9-whs2f\" (UID: \"950c3e6e-75a8-4e20-ba7f-84c0a08e1c1b\") " pod="openshift-ingress/e2e-api-a-8f95ee-openshift-default-5df565bcf9-whs2f" Jul 13 22:46:11.927471 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:46:11.927149 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/950c3e6e-75a8-4e20-ba7f-84c0a08e1c1b-istio-data\") pod \"e2e-api-a-8f95ee-openshift-default-5df565bcf9-whs2f\" (UID: \"950c3e6e-75a8-4e20-ba7f-84c0a08e1c1b\") " pod="openshift-ingress/e2e-api-a-8f95ee-openshift-default-5df565bcf9-whs2f" Jul 13 22:46:11.927471 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:46:11.927174 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/950c3e6e-75a8-4e20-ba7f-84c0a08e1c1b-istiod-ca-cert\") pod \"e2e-api-a-8f95ee-openshift-default-5df565bcf9-whs2f\" (UID: \"950c3e6e-75a8-4e20-ba7f-84c0a08e1c1b\") " pod="openshift-ingress/e2e-api-a-8f95ee-openshift-default-5df565bcf9-whs2f" Jul 13 22:46:11.927471 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:46:11.927204 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/950c3e6e-75a8-4e20-ba7f-84c0a08e1c1b-istio-envoy\") pod \"e2e-api-a-8f95ee-openshift-default-5df565bcf9-whs2f\" (UID: \"950c3e6e-75a8-4e20-ba7f-84c0a08e1c1b\") " pod="openshift-ingress/e2e-api-a-8f95ee-openshift-default-5df565bcf9-whs2f" Jul 13 22:46:11.927471 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:46:11.927272 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2f854\" (UniqueName: \"kubernetes.io/projected/950c3e6e-75a8-4e20-ba7f-84c0a08e1c1b-kube-api-access-2f854\") pod \"e2e-api-a-8f95ee-openshift-default-5df565bcf9-whs2f\" (UID: \"950c3e6e-75a8-4e20-ba7f-84c0a08e1c1b\") " pod="openshift-ingress/e2e-api-a-8f95ee-openshift-default-5df565bcf9-whs2f" Jul 13 22:46:11.927471 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:46:11.927314 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/950c3e6e-75a8-4e20-ba7f-84c0a08e1c1b-istio-token\") pod \"e2e-api-a-8f95ee-openshift-default-5df565bcf9-whs2f\" (UID: \"950c3e6e-75a8-4e20-ba7f-84c0a08e1c1b\") " pod="openshift-ingress/e2e-api-a-8f95ee-openshift-default-5df565bcf9-whs2f" Jul 13 22:46:11.927471 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:46:11.927452 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/950c3e6e-75a8-4e20-ba7f-84c0a08e1c1b-workload-socket\") pod \"e2e-api-a-8f95ee-openshift-default-5df565bcf9-whs2f\" (UID: \"950c3e6e-75a8-4e20-ba7f-84c0a08e1c1b\") " pod="openshift-ingress/e2e-api-a-8f95ee-openshift-default-5df565bcf9-whs2f" Jul 13 22:46:11.927991 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:46:11.927554 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/950c3e6e-75a8-4e20-ba7f-84c0a08e1c1b-credential-socket\") pod \"e2e-api-a-8f95ee-openshift-default-5df565bcf9-whs2f\" (UID: \"950c3e6e-75a8-4e20-ba7f-84c0a08e1c1b\") " pod="openshift-ingress/e2e-api-a-8f95ee-openshift-default-5df565bcf9-whs2f" Jul 13 22:46:11.927991 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:46:11.927563 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/950c3e6e-75a8-4e20-ba7f-84c0a08e1c1b-workload-certs\") pod \"e2e-api-a-8f95ee-openshift-default-5df565bcf9-whs2f\" (UID: \"950c3e6e-75a8-4e20-ba7f-84c0a08e1c1b\") " pod="openshift-ingress/e2e-api-a-8f95ee-openshift-default-5df565bcf9-whs2f" Jul 13 22:46:11.927991 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:46:11.927650 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/950c3e6e-75a8-4e20-ba7f-84c0a08e1c1b-istio-data\") pod \"e2e-api-a-8f95ee-openshift-default-5df565bcf9-whs2f\" (UID: \"950c3e6e-75a8-4e20-ba7f-84c0a08e1c1b\") " pod="openshift-ingress/e2e-api-a-8f95ee-openshift-default-5df565bcf9-whs2f" Jul 13 22:46:11.928154 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:46:11.928027 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/950c3e6e-75a8-4e20-ba7f-84c0a08e1c1b-istiod-ca-cert\") pod \"e2e-api-a-8f95ee-openshift-default-5df565bcf9-whs2f\" (UID: \"950c3e6e-75a8-4e20-ba7f-84c0a08e1c1b\") " pod="openshift-ingress/e2e-api-a-8f95ee-openshift-default-5df565bcf9-whs2f" Jul 13 22:46:11.929712 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:46:11.929688 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/950c3e6e-75a8-4e20-ba7f-84c0a08e1c1b-istio-podinfo\") pod \"e2e-api-a-8f95ee-openshift-default-5df565bcf9-whs2f\" (UID: \"950c3e6e-75a8-4e20-ba7f-84c0a08e1c1b\") " pod="openshift-ingress/e2e-api-a-8f95ee-openshift-default-5df565bcf9-whs2f" Jul 13 22:46:11.929853 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:46:11.929688 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/950c3e6e-75a8-4e20-ba7f-84c0a08e1c1b-istio-envoy\") pod \"e2e-api-a-8f95ee-openshift-default-5df565bcf9-whs2f\" (UID: \"950c3e6e-75a8-4e20-ba7f-84c0a08e1c1b\") " pod="openshift-ingress/e2e-api-a-8f95ee-openshift-default-5df565bcf9-whs2f" Jul 13 22:46:11.934761 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:46:11.934740 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/950c3e6e-75a8-4e20-ba7f-84c0a08e1c1b-istio-token\") pod \"e2e-api-a-8f95ee-openshift-default-5df565bcf9-whs2f\" (UID: \"950c3e6e-75a8-4e20-ba7f-84c0a08e1c1b\") " pod="openshift-ingress/e2e-api-a-8f95ee-openshift-default-5df565bcf9-whs2f" Jul 13 22:46:11.935100 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:46:11.935077 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2f854\" (UniqueName: \"kubernetes.io/projected/950c3e6e-75a8-4e20-ba7f-84c0a08e1c1b-kube-api-access-2f854\") pod \"e2e-api-a-8f95ee-openshift-default-5df565bcf9-whs2f\" (UID: \"950c3e6e-75a8-4e20-ba7f-84c0a08e1c1b\") " pod="openshift-ingress/e2e-api-a-8f95ee-openshift-default-5df565bcf9-whs2f" Jul 13 22:46:12.026098 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:46:12.026065 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-8f95ee-openshift-default-5df565bcf9-whs2f" Jul 13 22:46:12.164550 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:46:12.164525 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-8f95ee-openshift-default-5df565bcf9-whs2f"] Jul 13 22:46:12.166798 ip-10-0-133-179 kubenswrapper[2583]: W0713 22:46:12.166773 2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod950c3e6e_75a8_4e20_ba7f_84c0a08e1c1b.slice/crio-236baadd5c7958f0da41a7cadb92644c89352c3da2e2eb1739131fa1e29d241e WatchSource:0}: Error finding container 236baadd5c7958f0da41a7cadb92644c89352c3da2e2eb1739131fa1e29d241e: Status 404 returned error can't find the container with id 236baadd5c7958f0da41a7cadb92644c89352c3da2e2eb1739131fa1e29d241e Jul 13 22:46:12.169015 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:46:12.168977 2583 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 13 22:46:12.169100 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:46:12.169049 2583 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 13 22:46:12.169100 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:46:12.169087 2583 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 13 22:46:12.343435 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:46:12.343351 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-8f95ee-openshift-default-5df565bcf9-whs2f" event={"ID":"950c3e6e-75a8-4e20-ba7f-84c0a08e1c1b","Type":"ContainerStarted","Data":"f4251a7a13c3037c580f09edee54ea25330c989784a5480395bd3cfa3b74ca51"} Jul 13 22:46:12.343435 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:46:12.343407 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-8f95ee-openshift-default-5df565bcf9-whs2f" event={"ID":"950c3e6e-75a8-4e20-ba7f-84c0a08e1c1b","Type":"ContainerStarted","Data":"236baadd5c7958f0da41a7cadb92644c89352c3da2e2eb1739131fa1e29d241e"} Jul 13 22:46:12.361224 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:46:12.361178 2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-api-a-8f95ee-openshift-default-5df565bcf9-whs2f" podStartSLOduration=1.361158149 podStartE2EDuration="1.361158149s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-13 22:46:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 22:46:12.359948257 +0000 UTC m=+1967.607863606" watchObservedRunningTime="2026-07-13 22:46:12.361158149 +0000 UTC m=+1967.609073503" Jul 13 22:46:13.026747 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:46:13.026706 2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-8f95ee-openshift-default-5df565bcf9-whs2f" Jul 13 22:46:13.031692 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:46:13.031664 2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-8f95ee-openshift-default-5df565bcf9-whs2f" Jul 13 22:46:13.347254 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:46:13.347164 2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-8f95ee-openshift-default-5df565bcf9-whs2f" Jul 13 22:46:13.348254 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:46:13.348234 2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-8f95ee-openshift-default-5df565bcf9-whs2f" Jul 13 22:46:14.738985 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:46:14.738937 2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-8f95ee-6779c6cb5c-rkjwz"] Jul 13 22:46:14.742604 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:46:14.742582 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-8f95ee-6779c6cb5c-rkjwz" Jul 13 22:46:14.744732 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:46:14.744705 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-a-8f95ee-serving-cert\"" Jul 13 22:46:14.753611 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:46:14.753588 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-8f95ee-6779c6cb5c-rkjwz"] Jul 13 22:46:14.853012 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:46:14.852968 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/dadfd181-e4c0-4f74-841f-fc644964cbd1-openshift-service-ca\") pod \"maas-api-e2e-api-a-8f95ee-6779c6cb5c-rkjwz\" (UID: \"dadfd181-e4c0-4f74-841f-fc644964cbd1\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-8f95ee-6779c6cb5c-rkjwz" Jul 13 22:46:14.853209 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:46:14.853055 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n46vj\" (UniqueName: \"kubernetes.io/projected/dadfd181-e4c0-4f74-841f-fc644964cbd1-kube-api-access-n46vj\") pod \"maas-api-e2e-api-a-8f95ee-6779c6cb5c-rkjwz\" (UID: \"dadfd181-e4c0-4f74-841f-fc644964cbd1\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-8f95ee-6779c6cb5c-rkjwz" Jul 13 22:46:14.853209 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:46:14.853084 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/dadfd181-e4c0-4f74-841f-fc644964cbd1-maas-api-tls\") pod \"maas-api-e2e-api-a-8f95ee-6779c6cb5c-rkjwz\" (UID: \"dadfd181-e4c0-4f74-841f-fc644964cbd1\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-8f95ee-6779c6cb5c-rkjwz" Jul 13 22:46:14.953788 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:46:14.953742 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n46vj\" (UniqueName: \"kubernetes.io/projected/dadfd181-e4c0-4f74-841f-fc644964cbd1-kube-api-access-n46vj\") pod \"maas-api-e2e-api-a-8f95ee-6779c6cb5c-rkjwz\" (UID: \"dadfd181-e4c0-4f74-841f-fc644964cbd1\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-8f95ee-6779c6cb5c-rkjwz" Jul 13 22:46:14.953990 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:46:14.953797 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/dadfd181-e4c0-4f74-841f-fc644964cbd1-maas-api-tls\") pod \"maas-api-e2e-api-a-8f95ee-6779c6cb5c-rkjwz\" (UID: \"dadfd181-e4c0-4f74-841f-fc644964cbd1\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-8f95ee-6779c6cb5c-rkjwz" Jul 13 22:46:14.953990 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:46:14.953855 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/dadfd181-e4c0-4f74-841f-fc644964cbd1-openshift-service-ca\") pod \"maas-api-e2e-api-a-8f95ee-6779c6cb5c-rkjwz\" (UID: \"dadfd181-e4c0-4f74-841f-fc644964cbd1\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-8f95ee-6779c6cb5c-rkjwz" Jul 13 22:46:14.954533 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:46:14.954506 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/dadfd181-e4c0-4f74-841f-fc644964cbd1-openshift-service-ca\") pod \"maas-api-e2e-api-a-8f95ee-6779c6cb5c-rkjwz\" (UID: \"dadfd181-e4c0-4f74-841f-fc644964cbd1\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-8f95ee-6779c6cb5c-rkjwz" Jul 13 22:46:14.956271 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:46:14.956250 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/dadfd181-e4c0-4f74-841f-fc644964cbd1-maas-api-tls\") pod \"maas-api-e2e-api-a-8f95ee-6779c6cb5c-rkjwz\" (UID: \"dadfd181-e4c0-4f74-841f-fc644964cbd1\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-8f95ee-6779c6cb5c-rkjwz" Jul 13 22:46:14.962937 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:46:14.962916 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-n46vj\" (UniqueName: \"kubernetes.io/projected/dadfd181-e4c0-4f74-841f-fc644964cbd1-kube-api-access-n46vj\") pod \"maas-api-e2e-api-a-8f95ee-6779c6cb5c-rkjwz\" (UID: \"dadfd181-e4c0-4f74-841f-fc644964cbd1\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-8f95ee-6779c6cb5c-rkjwz" Jul 13 22:46:15.056329 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:46:15.056298 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-8f95ee-6779c6cb5c-rkjwz" Jul 13 22:46:15.207223 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:46:15.207194 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-8f95ee-6779c6cb5c-rkjwz"] Jul 13 22:46:15.210134 ip-10-0-133-179 kubenswrapper[2583]: W0713 22:46:15.210099 2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddadfd181_e4c0_4f74_841f_fc644964cbd1.slice/crio-09ae8980792ca7729910901678e150d02e2d8ff02a273752eae0fd1d34950120 WatchSource:0}: Error finding container 09ae8980792ca7729910901678e150d02e2d8ff02a273752eae0fd1d34950120: Status 404 returned error can't find the container with id 09ae8980792ca7729910901678e150d02e2d8ff02a273752eae0fd1d34950120 Jul 13 22:46:15.357002 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:46:15.356917 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-8f95ee-6779c6cb5c-rkjwz" event={"ID":"dadfd181-e4c0-4f74-841f-fc644964cbd1","Type":"ContainerStarted","Data":"09ae8980792ca7729910901678e150d02e2d8ff02a273752eae0fd1d34950120"} Jul 13 22:46:16.074387 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:46:16.074362 2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-953e38-openshift-default-5567878d9c-r76gn" Jul 13 22:46:16.164627 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:46:16.164588 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d9c803a7-01f3-409c-857f-48b2246e009f-istio-podinfo\") pod \"d9c803a7-01f3-409c-857f-48b2246e009f\" (UID: \"d9c803a7-01f3-409c-857f-48b2246e009f\") " Jul 13 22:46:16.164819 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:46:16.164677 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d9c803a7-01f3-409c-857f-48b2246e009f-istio-token\") pod \"d9c803a7-01f3-409c-857f-48b2246e009f\" (UID: \"d9c803a7-01f3-409c-857f-48b2246e009f\") " Jul 13 22:46:16.164819 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:46:16.164713 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d9c803a7-01f3-409c-857f-48b2246e009f-istio-envoy\") pod \"d9c803a7-01f3-409c-857f-48b2246e009f\" (UID: \"d9c803a7-01f3-409c-857f-48b2246e009f\") " Jul 13 22:46:16.164819 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:46:16.164752 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d9c803a7-01f3-409c-857f-48b2246e009f-istiod-ca-cert\") pod \"d9c803a7-01f3-409c-857f-48b2246e009f\" (UID: \"d9c803a7-01f3-409c-857f-48b2246e009f\") " Jul 13 22:46:16.164819 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:46:16.164804 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d9c803a7-01f3-409c-857f-48b2246e009f-workload-certs\") pod \"d9c803a7-01f3-409c-857f-48b2246e009f\" (UID: \"d9c803a7-01f3-409c-857f-48b2246e009f\") " Jul 13 22:46:16.165035 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:46:16.164833 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d9c803a7-01f3-409c-857f-48b2246e009f-workload-socket\") pod \"d9c803a7-01f3-409c-857f-48b2246e009f\" (UID: \"d9c803a7-01f3-409c-857f-48b2246e009f\") " Jul 13 22:46:16.165035 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:46:16.164858 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2jfvp\" (UniqueName: \"kubernetes.io/projected/d9c803a7-01f3-409c-857f-48b2246e009f-kube-api-access-2jfvp\") pod \"d9c803a7-01f3-409c-857f-48b2246e009f\" (UID: \"d9c803a7-01f3-409c-857f-48b2246e009f\") " Jul 13 22:46:16.165035 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:46:16.164893 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d9c803a7-01f3-409c-857f-48b2246e009f-istio-data\") pod \"d9c803a7-01f3-409c-857f-48b2246e009f\" (UID: \"d9c803a7-01f3-409c-857f-48b2246e009f\") " Jul 13 22:46:16.165035 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:46:16.164931 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d9c803a7-01f3-409c-857f-48b2246e009f-credential-socket\") pod \"d9c803a7-01f3-409c-857f-48b2246e009f\" (UID: \"d9c803a7-01f3-409c-857f-48b2246e009f\") " Jul 13 22:46:16.165218 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:46:16.165145 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d9c803a7-01f3-409c-857f-48b2246e009f-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "d9c803a7-01f3-409c-857f-48b2246e009f" (UID: "d9c803a7-01f3-409c-857f-48b2246e009f"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 13 22:46:16.165218 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:46:16.165162 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d9c803a7-01f3-409c-857f-48b2246e009f-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "d9c803a7-01f3-409c-857f-48b2246e009f" (UID: "d9c803a7-01f3-409c-857f-48b2246e009f"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 13 22:46:16.165461 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:46:16.165422 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d9c803a7-01f3-409c-857f-48b2246e009f-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "d9c803a7-01f3-409c-857f-48b2246e009f" (UID: "d9c803a7-01f3-409c-857f-48b2246e009f"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 13 22:46:16.165461 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:46:16.165432 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d9c803a7-01f3-409c-857f-48b2246e009f-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "d9c803a7-01f3-409c-857f-48b2246e009f" (UID: "d9c803a7-01f3-409c-857f-48b2246e009f"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 13 22:46:16.166121 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:46:16.166097 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d9c803a7-01f3-409c-857f-48b2246e009f-istio-data" (OuterVolumeSpecName: "istio-data") pod "d9c803a7-01f3-409c-857f-48b2246e009f" (UID: "d9c803a7-01f3-409c-857f-48b2246e009f"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 13 22:46:16.166909 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:46:16.166885 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d9c803a7-01f3-409c-857f-48b2246e009f-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "d9c803a7-01f3-409c-857f-48b2246e009f" (UID: "d9c803a7-01f3-409c-857f-48b2246e009f"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 13 22:46:16.167030 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:46:16.167005 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/d9c803a7-01f3-409c-857f-48b2246e009f-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "d9c803a7-01f3-409c-857f-48b2246e009f" (UID: "d9c803a7-01f3-409c-857f-48b2246e009f"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 13 22:46:16.167219 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:46:16.167195 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d9c803a7-01f3-409c-857f-48b2246e009f-istio-token" (OuterVolumeSpecName: "istio-token") pod "d9c803a7-01f3-409c-857f-48b2246e009f" (UID: "d9c803a7-01f3-409c-857f-48b2246e009f"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 13 22:46:16.167324 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:46:16.167298 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d9c803a7-01f3-409c-857f-48b2246e009f-kube-api-access-2jfvp" (OuterVolumeSpecName: "kube-api-access-2jfvp") pod "d9c803a7-01f3-409c-857f-48b2246e009f" (UID: "d9c803a7-01f3-409c-857f-48b2246e009f"). InnerVolumeSpecName "kube-api-access-2jfvp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 13 22:46:16.266282 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:46:16.266239 2583 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d9c803a7-01f3-409c-857f-48b2246e009f-workload-certs\") on node \"ip-10-0-133-179.ec2.internal\" DevicePath \"\"" Jul 13 22:46:16.266282 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:46:16.266271 2583 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d9c803a7-01f3-409c-857f-48b2246e009f-workload-socket\") on node \"ip-10-0-133-179.ec2.internal\" DevicePath \"\"" Jul 13 22:46:16.266282 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:46:16.266281 2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2jfvp\" (UniqueName: \"kubernetes.io/projected/d9c803a7-01f3-409c-857f-48b2246e009f-kube-api-access-2jfvp\") on node \"ip-10-0-133-179.ec2.internal\" DevicePath \"\"" Jul 13 22:46:16.266282 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:46:16.266291 2583 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d9c803a7-01f3-409c-857f-48b2246e009f-istio-data\") on node \"ip-10-0-133-179.ec2.internal\" DevicePath \"\"" Jul 13 22:46:16.266539 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:46:16.266300 2583 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d9c803a7-01f3-409c-857f-48b2246e009f-credential-socket\") on node \"ip-10-0-133-179.ec2.internal\" DevicePath \"\"" Jul 13 22:46:16.266539 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:46:16.266308 2583 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d9c803a7-01f3-409c-857f-48b2246e009f-istio-podinfo\") on node \"ip-10-0-133-179.ec2.internal\" DevicePath \"\"" Jul 13 22:46:16.266539 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:46:16.266318 2583 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d9c803a7-01f3-409c-857f-48b2246e009f-istio-token\") on node \"ip-10-0-133-179.ec2.internal\" DevicePath \"\"" Jul 13 22:46:16.266539 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:46:16.266326 2583 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d9c803a7-01f3-409c-857f-48b2246e009f-istio-envoy\") on node \"ip-10-0-133-179.ec2.internal\" DevicePath \"\"" Jul 13 22:46:16.266539 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:46:16.266334 2583 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d9c803a7-01f3-409c-857f-48b2246e009f-istiod-ca-cert\") on node \"ip-10-0-133-179.ec2.internal\" DevicePath \"\"" Jul 13 22:46:16.362626 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:46:16.362522 2583 generic.go:356] "Generic (PLEG): container finished" podID="d9c803a7-01f3-409c-857f-48b2246e009f" containerID="6220c761ef6d6a8ea40e5cd28f8e12841177b01692c3bf2a3a5984d0c7fa1c27" exitCode=0 Jul 13 22:46:16.362626 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:46:16.362594 2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-953e38-openshift-default-5567878d9c-r76gn" Jul 13 22:46:16.362819 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:46:16.362632 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-953e38-openshift-default-5567878d9c-r76gn" event={"ID":"d9c803a7-01f3-409c-857f-48b2246e009f","Type":"ContainerDied","Data":"6220c761ef6d6a8ea40e5cd28f8e12841177b01692c3bf2a3a5984d0c7fa1c27"} Jul 13 22:46:16.362819 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:46:16.362687 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-953e38-openshift-default-5567878d9c-r76gn" event={"ID":"d9c803a7-01f3-409c-857f-48b2246e009f","Type":"ContainerDied","Data":"1ed466a7c4c40bc859ba2019eae404cbc05bc9a52b5311dbceb0a2cd57154965"} Jul 13 22:46:16.362819 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:46:16.362713 2583 scope.go:117] "RemoveContainer" containerID="6220c761ef6d6a8ea40e5cd28f8e12841177b01692c3bf2a3a5984d0c7fa1c27" Jul 13 22:46:16.364231 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:46:16.364209 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-8f95ee-6779c6cb5c-rkjwz" event={"ID":"dadfd181-e4c0-4f74-841f-fc644964cbd1","Type":"ContainerStarted","Data":"50af117b02ca2c10c4098747ff3e31b7b8898994292a0eeb98e95b4ab82c96c6"} Jul 13 22:46:16.364307 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:46:16.364272 2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-8f95ee-6779c6cb5c-rkjwz" Jul 13 22:46:16.371893 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:46:16.371873 2583 scope.go:117] "RemoveContainer" containerID="d98537e258409ec0ba624123662fc645088a09e3bf6207517e40d9f050e37816" Jul 13 22:46:16.379220 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:46:16.379205 2583 scope.go:117] "RemoveContainer" containerID="6220c761ef6d6a8ea40e5cd28f8e12841177b01692c3bf2a3a5984d0c7fa1c27" Jul 13 22:46:16.379456 ip-10-0-133-179 kubenswrapper[2583]: E0713 22:46:16.379441 2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6220c761ef6d6a8ea40e5cd28f8e12841177b01692c3bf2a3a5984d0c7fa1c27\": container with ID starting with 6220c761ef6d6a8ea40e5cd28f8e12841177b01692c3bf2a3a5984d0c7fa1c27 not found: ID does not exist" containerID="6220c761ef6d6a8ea40e5cd28f8e12841177b01692c3bf2a3a5984d0c7fa1c27" Jul 13 22:46:16.379513 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:46:16.379463 2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6220c761ef6d6a8ea40e5cd28f8e12841177b01692c3bf2a3a5984d0c7fa1c27"} err="failed to get container status \"6220c761ef6d6a8ea40e5cd28f8e12841177b01692c3bf2a3a5984d0c7fa1c27\": rpc error: code = NotFound desc = could not find container \"6220c761ef6d6a8ea40e5cd28f8e12841177b01692c3bf2a3a5984d0c7fa1c27\": container with ID starting with 6220c761ef6d6a8ea40e5cd28f8e12841177b01692c3bf2a3a5984d0c7fa1c27 not found: ID does not exist" Jul 13 22:46:16.379513 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:46:16.379481 2583 scope.go:117] "RemoveContainer" containerID="d98537e258409ec0ba624123662fc645088a09e3bf6207517e40d9f050e37816" Jul 13 22:46:16.379722 ip-10-0-133-179 kubenswrapper[2583]: E0713 22:46:16.379707 2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d98537e258409ec0ba624123662fc645088a09e3bf6207517e40d9f050e37816\": container with ID starting with d98537e258409ec0ba624123662fc645088a09e3bf6207517e40d9f050e37816 not found: ID does not exist" containerID="d98537e258409ec0ba624123662fc645088a09e3bf6207517e40d9f050e37816" Jul 13 22:46:16.379770 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:46:16.379725 2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d98537e258409ec0ba624123662fc645088a09e3bf6207517e40d9f050e37816"} err="failed to get container status \"d98537e258409ec0ba624123662fc645088a09e3bf6207517e40d9f050e37816\": rpc error: code = NotFound desc = could not find container \"d98537e258409ec0ba624123662fc645088a09e3bf6207517e40d9f050e37816\": container with ID starting with d98537e258409ec0ba624123662fc645088a09e3bf6207517e40d9f050e37816 not found: ID does not exist" Jul 13 22:46:16.385690 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:46:16.385629 2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-8f95ee-6779c6cb5c-rkjwz" podStartSLOduration=1.8016525209999998 podStartE2EDuration="2.385612999s" totalImagesPullingTime="583.960478ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-13 22:46:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 22:46:16.385302238 +0000 UTC m=+1971.633217601" watchObservedRunningTime="2026-07-13 22:46:16.385612999 +0000 UTC m=+1971.633528350" Jul 13 22:46:16.398303 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:46:16.398284 2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-953e38-openshift-default-5567878d9c-r76gn"] Jul 13 22:46:16.403556 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:46:16.403538 2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-b-953e38-openshift-default-5567878d9c-r76gn"] Jul 13 22:46:17.390225 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:46:17.390190 2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d9c803a7-01f3-409c-857f-48b2246e009f" path="/var/lib/kubelet/pods/d9c803a7-01f3-409c-857f-48b2246e009f/volumes" Jul 13 22:46:19.497853 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:46:19.497819 2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-b-1276a7-openshift-default-84f5889fb9-2qzdr"] Jul 13 22:46:19.498195 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:46:19.498154 2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d9c803a7-01f3-409c-857f-48b2246e009f" containerName="istio-proxy" Jul 13 22:46:19.498195 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:46:19.498164 2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="d9c803a7-01f3-409c-857f-48b2246e009f" containerName="istio-proxy" Jul 13 22:46:19.498319 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:46:19.498217 2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="d9c803a7-01f3-409c-857f-48b2246e009f" containerName="istio-proxy" Jul 13 22:46:19.502644 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:46:19.502624 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-1276a7-openshift-default-84f5889fb9-2qzdr" Jul 13 22:46:19.505177 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:46:19.505151 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-b-1276a7-openshift-default-dockercfg-fjqrh\"" Jul 13 22:46:19.513783 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:46:19.513751 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-1276a7-openshift-default-84f5889fb9-2qzdr"] Jul 13 22:46:19.597259 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:46:19.597220 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0b5a60d1-133b-4e0b-b868-fbd4adc706db-istio-podinfo\") pod \"e2e-api-b-1276a7-openshift-default-84f5889fb9-2qzdr\" (UID: \"0b5a60d1-133b-4e0b-b868-fbd4adc706db\") " pod="openshift-ingress/e2e-api-b-1276a7-openshift-default-84f5889fb9-2qzdr" Jul 13 22:46:19.597259 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:46:19.597267 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qdpwj\" (UniqueName: \"kubernetes.io/projected/0b5a60d1-133b-4e0b-b868-fbd4adc706db-kube-api-access-qdpwj\") pod \"e2e-api-b-1276a7-openshift-default-84f5889fb9-2qzdr\" (UID: \"0b5a60d1-133b-4e0b-b868-fbd4adc706db\") " pod="openshift-ingress/e2e-api-b-1276a7-openshift-default-84f5889fb9-2qzdr" Jul 13 22:46:19.597505 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:46:19.597311 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0b5a60d1-133b-4e0b-b868-fbd4adc706db-istiod-ca-cert\") pod \"e2e-api-b-1276a7-openshift-default-84f5889fb9-2qzdr\" (UID: \"0b5a60d1-133b-4e0b-b868-fbd4adc706db\") " pod="openshift-ingress/e2e-api-b-1276a7-openshift-default-84f5889fb9-2qzdr" Jul 13 22:46:19.597505 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:46:19.597354 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0b5a60d1-133b-4e0b-b868-fbd4adc706db-istio-token\") pod \"e2e-api-b-1276a7-openshift-default-84f5889fb9-2qzdr\" (UID: \"0b5a60d1-133b-4e0b-b868-fbd4adc706db\") " pod="openshift-ingress/e2e-api-b-1276a7-openshift-default-84f5889fb9-2qzdr" Jul 13 22:46:19.597505 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:46:19.597381 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0b5a60d1-133b-4e0b-b868-fbd4adc706db-workload-certs\") pod \"e2e-api-b-1276a7-openshift-default-84f5889fb9-2qzdr\" (UID: \"0b5a60d1-133b-4e0b-b868-fbd4adc706db\") " pod="openshift-ingress/e2e-api-b-1276a7-openshift-default-84f5889fb9-2qzdr" Jul 13 22:46:19.597505 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:46:19.597419 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0b5a60d1-133b-4e0b-b868-fbd4adc706db-istio-data\") pod \"e2e-api-b-1276a7-openshift-default-84f5889fb9-2qzdr\" (UID: \"0b5a60d1-133b-4e0b-b868-fbd4adc706db\") " pod="openshift-ingress/e2e-api-b-1276a7-openshift-default-84f5889fb9-2qzdr" Jul 13 22:46:19.597505 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:46:19.597456 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0b5a60d1-133b-4e0b-b868-fbd4adc706db-istio-envoy\") pod \"e2e-api-b-1276a7-openshift-default-84f5889fb9-2qzdr\" (UID: \"0b5a60d1-133b-4e0b-b868-fbd4adc706db\") " pod="openshift-ingress/e2e-api-b-1276a7-openshift-default-84f5889fb9-2qzdr" Jul 13 22:46:19.597505 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:46:19.597498 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0b5a60d1-133b-4e0b-b868-fbd4adc706db-workload-socket\") pod \"e2e-api-b-1276a7-openshift-default-84f5889fb9-2qzdr\" (UID: \"0b5a60d1-133b-4e0b-b868-fbd4adc706db\") " pod="openshift-ingress/e2e-api-b-1276a7-openshift-default-84f5889fb9-2qzdr" Jul 13 22:46:19.597843 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:46:19.597524 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0b5a60d1-133b-4e0b-b868-fbd4adc706db-credential-socket\") pod \"e2e-api-b-1276a7-openshift-default-84f5889fb9-2qzdr\" (UID: \"0b5a60d1-133b-4e0b-b868-fbd4adc706db\") " pod="openshift-ingress/e2e-api-b-1276a7-openshift-default-84f5889fb9-2qzdr" Jul 13 22:46:19.698248 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:46:19.698210 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0b5a60d1-133b-4e0b-b868-fbd4adc706db-credential-socket\") pod \"e2e-api-b-1276a7-openshift-default-84f5889fb9-2qzdr\" (UID: \"0b5a60d1-133b-4e0b-b868-fbd4adc706db\") " pod="openshift-ingress/e2e-api-b-1276a7-openshift-default-84f5889fb9-2qzdr" Jul 13 22:46:19.698410 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:46:19.698267 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0b5a60d1-133b-4e0b-b868-fbd4adc706db-istio-podinfo\") pod \"e2e-api-b-1276a7-openshift-default-84f5889fb9-2qzdr\" (UID: \"0b5a60d1-133b-4e0b-b868-fbd4adc706db\") " pod="openshift-ingress/e2e-api-b-1276a7-openshift-default-84f5889fb9-2qzdr" Jul 13 22:46:19.698410 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:46:19.698296 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qdpwj\" (UniqueName: \"kubernetes.io/projected/0b5a60d1-133b-4e0b-b868-fbd4adc706db-kube-api-access-qdpwj\") pod \"e2e-api-b-1276a7-openshift-default-84f5889fb9-2qzdr\" (UID: \"0b5a60d1-133b-4e0b-b868-fbd4adc706db\") " pod="openshift-ingress/e2e-api-b-1276a7-openshift-default-84f5889fb9-2qzdr" Jul 13 22:46:19.698410 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:46:19.698339 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0b5a60d1-133b-4e0b-b868-fbd4adc706db-istiod-ca-cert\") pod \"e2e-api-b-1276a7-openshift-default-84f5889fb9-2qzdr\" (UID: \"0b5a60d1-133b-4e0b-b868-fbd4adc706db\") " pod="openshift-ingress/e2e-api-b-1276a7-openshift-default-84f5889fb9-2qzdr" Jul 13 22:46:19.698410 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:46:19.698381 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0b5a60d1-133b-4e0b-b868-fbd4adc706db-istio-token\") pod \"e2e-api-b-1276a7-openshift-default-84f5889fb9-2qzdr\" (UID: \"0b5a60d1-133b-4e0b-b868-fbd4adc706db\") " pod="openshift-ingress/e2e-api-b-1276a7-openshift-default-84f5889fb9-2qzdr" Jul 13 22:46:19.698410 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:46:19.698407 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0b5a60d1-133b-4e0b-b868-fbd4adc706db-workload-certs\") pod \"e2e-api-b-1276a7-openshift-default-84f5889fb9-2qzdr\" (UID: \"0b5a60d1-133b-4e0b-b868-fbd4adc706db\") " pod="openshift-ingress/e2e-api-b-1276a7-openshift-default-84f5889fb9-2qzdr" Jul 13 22:46:19.698708 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:46:19.698448 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0b5a60d1-133b-4e0b-b868-fbd4adc706db-istio-data\") pod \"e2e-api-b-1276a7-openshift-default-84f5889fb9-2qzdr\" (UID: \"0b5a60d1-133b-4e0b-b868-fbd4adc706db\") " pod="openshift-ingress/e2e-api-b-1276a7-openshift-default-84f5889fb9-2qzdr" Jul 13 22:46:19.698708 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:46:19.698491 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0b5a60d1-133b-4e0b-b868-fbd4adc706db-istio-envoy\") pod \"e2e-api-b-1276a7-openshift-default-84f5889fb9-2qzdr\" (UID: \"0b5a60d1-133b-4e0b-b868-fbd4adc706db\") " pod="openshift-ingress/e2e-api-b-1276a7-openshift-default-84f5889fb9-2qzdr" Jul 13 22:46:19.698708 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:46:19.698538 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0b5a60d1-133b-4e0b-b868-fbd4adc706db-workload-socket\") pod \"e2e-api-b-1276a7-openshift-default-84f5889fb9-2qzdr\" (UID: \"0b5a60d1-133b-4e0b-b868-fbd4adc706db\") " pod="openshift-ingress/e2e-api-b-1276a7-openshift-default-84f5889fb9-2qzdr" Jul 13 22:46:19.698708 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:46:19.698679 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0b5a60d1-133b-4e0b-b868-fbd4adc706db-credential-socket\") pod \"e2e-api-b-1276a7-openshift-default-84f5889fb9-2qzdr\" (UID: \"0b5a60d1-133b-4e0b-b868-fbd4adc706db\") " pod="openshift-ingress/e2e-api-b-1276a7-openshift-default-84f5889fb9-2qzdr" Jul 13 22:46:19.698932 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:46:19.698856 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0b5a60d1-133b-4e0b-b868-fbd4adc706db-workload-socket\") pod \"e2e-api-b-1276a7-openshift-default-84f5889fb9-2qzdr\" (UID: \"0b5a60d1-133b-4e0b-b868-fbd4adc706db\") " pod="openshift-ingress/e2e-api-b-1276a7-openshift-default-84f5889fb9-2qzdr" Jul 13 22:46:19.698985 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:46:19.698938 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0b5a60d1-133b-4e0b-b868-fbd4adc706db-workload-certs\") pod \"e2e-api-b-1276a7-openshift-default-84f5889fb9-2qzdr\" (UID: \"0b5a60d1-133b-4e0b-b868-fbd4adc706db\") " pod="openshift-ingress/e2e-api-b-1276a7-openshift-default-84f5889fb9-2qzdr" Jul 13 22:46:19.699097 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:46:19.699077 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0b5a60d1-133b-4e0b-b868-fbd4adc706db-istio-data\") pod \"e2e-api-b-1276a7-openshift-default-84f5889fb9-2qzdr\" (UID: \"0b5a60d1-133b-4e0b-b868-fbd4adc706db\") " pod="openshift-ingress/e2e-api-b-1276a7-openshift-default-84f5889fb9-2qzdr" Jul 13 22:46:19.699155 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:46:19.699095 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0b5a60d1-133b-4e0b-b868-fbd4adc706db-istiod-ca-cert\") pod \"e2e-api-b-1276a7-openshift-default-84f5889fb9-2qzdr\" (UID: \"0b5a60d1-133b-4e0b-b868-fbd4adc706db\") " pod="openshift-ingress/e2e-api-b-1276a7-openshift-default-84f5889fb9-2qzdr" Jul 13 22:46:19.700861 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:46:19.700837 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0b5a60d1-133b-4e0b-b868-fbd4adc706db-istio-envoy\") pod \"e2e-api-b-1276a7-openshift-default-84f5889fb9-2qzdr\" (UID: \"0b5a60d1-133b-4e0b-b868-fbd4adc706db\") " pod="openshift-ingress/e2e-api-b-1276a7-openshift-default-84f5889fb9-2qzdr" Jul 13 22:46:19.700967 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:46:19.700898 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0b5a60d1-133b-4e0b-b868-fbd4adc706db-istio-podinfo\") pod \"e2e-api-b-1276a7-openshift-default-84f5889fb9-2qzdr\" (UID: \"0b5a60d1-133b-4e0b-b868-fbd4adc706db\") " pod="openshift-ingress/e2e-api-b-1276a7-openshift-default-84f5889fb9-2qzdr" Jul 13 22:46:19.711000 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:46:19.710971 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0b5a60d1-133b-4e0b-b868-fbd4adc706db-istio-token\") pod \"e2e-api-b-1276a7-openshift-default-84f5889fb9-2qzdr\" (UID: \"0b5a60d1-133b-4e0b-b868-fbd4adc706db\") " pod="openshift-ingress/e2e-api-b-1276a7-openshift-default-84f5889fb9-2qzdr" Jul 13 22:46:19.711183 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:46:19.711163 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qdpwj\" (UniqueName: \"kubernetes.io/projected/0b5a60d1-133b-4e0b-b868-fbd4adc706db-kube-api-access-qdpwj\") pod \"e2e-api-b-1276a7-openshift-default-84f5889fb9-2qzdr\" (UID: \"0b5a60d1-133b-4e0b-b868-fbd4adc706db\") " pod="openshift-ingress/e2e-api-b-1276a7-openshift-default-84f5889fb9-2qzdr" Jul 13 22:46:19.813663 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:46:19.813627 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-1276a7-openshift-default-84f5889fb9-2qzdr" Jul 13 22:46:19.952859 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:46:19.952833 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-1276a7-openshift-default-84f5889fb9-2qzdr"] Jul 13 22:46:19.954360 ip-10-0-133-179 kubenswrapper[2583]: W0713 22:46:19.954330 2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0b5a60d1_133b_4e0b_b868_fbd4adc706db.slice/crio-3fb5a8a7164f11e3ce1b914e8a7f3cb30b5ecbb5cca71083f6761d8966662371 WatchSource:0}: Error finding container 3fb5a8a7164f11e3ce1b914e8a7f3cb30b5ecbb5cca71083f6761d8966662371: Status 404 returned error can't find the container with id 3fb5a8a7164f11e3ce1b914e8a7f3cb30b5ecbb5cca71083f6761d8966662371 Jul 13 22:46:19.956624 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:46:19.956588 2583 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 13 22:46:19.956695 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:46:19.956654 2583 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 13 22:46:19.956695 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:46:19.956683 2583 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 13 22:46:20.384506 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:46:20.384470 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-1276a7-openshift-default-84f5889fb9-2qzdr" event={"ID":"0b5a60d1-133b-4e0b-b868-fbd4adc706db","Type":"ContainerStarted","Data":"b98188f01ac9ee00971d7d1060c4934b04e42a7424cc8cf322450e6cf55c4dff"} Jul 13 22:46:20.384804 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:46:20.384523 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-1276a7-openshift-default-84f5889fb9-2qzdr" event={"ID":"0b5a60d1-133b-4e0b-b868-fbd4adc706db","Type":"ContainerStarted","Data":"3fb5a8a7164f11e3ce1b914e8a7f3cb30b5ecbb5cca71083f6761d8966662371"} Jul 13 22:46:20.406816 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:46:20.406764 2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-api-b-1276a7-openshift-default-84f5889fb9-2qzdr" podStartSLOduration=1.4067467040000001 podStartE2EDuration="1.406746704s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-13 22:46:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 22:46:20.405077247 +0000 UTC m=+1975.652992607" watchObservedRunningTime="2026-07-13 22:46:20.406746704 +0000 UTC m=+1975.654662127" Jul 13 22:46:20.814142 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:46:20.814109 2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-1276a7-openshift-default-84f5889fb9-2qzdr" Jul 13 22:46:21.819453 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:46:21.819420 2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-1276a7-openshift-default-84f5889fb9-2qzdr" Jul 13 22:46:22.376923 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:46:22.376898 2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-8f95ee-6779c6cb5c-rkjwz" Jul 13 22:46:22.391921 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:46:22.391890 2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-1276a7-openshift-default-84f5889fb9-2qzdr" Jul 13 22:46:22.392774 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:46:22.392753 2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-1276a7-openshift-default-84f5889fb9-2qzdr" Jul 13 22:46:22.457916 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:46:22.457865 2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-1276a7-5b54c55668-2lgc6"] Jul 13 22:46:22.464213 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:46:22.464177 2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d9c803a7-01f3-409c-857f-48b2246e009f" containerName="istio-proxy" Jul 13 22:46:22.464406 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:46:22.464392 2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="d9c803a7-01f3-409c-857f-48b2246e009f" containerName="istio-proxy" Jul 13 22:46:22.468499 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:46:22.468475 2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="d9c803a7-01f3-409c-857f-48b2246e009f" containerName="istio-proxy" Jul 13 22:46:22.471943 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:46:22.471919 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-1276a7-5b54c55668-2lgc6"] Jul 13 22:46:22.472051 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:46:22.472014 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-1276a7-5b54c55668-2lgc6" Jul 13 22:46:22.474052 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:46:22.474030 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-b-1276a7-serving-cert\"" Jul 13 22:46:22.524335 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:46:22.524298 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9a4f8fb1-fb22-4a74-a2b9-e89938780f62-maas-api-tls\") pod \"maas-api-e2e-api-b-1276a7-5b54c55668-2lgc6\" (UID: \"9a4f8fb1-fb22-4a74-a2b9-e89938780f62\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-1276a7-5b54c55668-2lgc6" Jul 13 22:46:22.524509 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:46:22.524358 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9a4f8fb1-fb22-4a74-a2b9-e89938780f62-openshift-service-ca\") pod \"maas-api-e2e-api-b-1276a7-5b54c55668-2lgc6\" (UID: \"9a4f8fb1-fb22-4a74-a2b9-e89938780f62\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-1276a7-5b54c55668-2lgc6" Jul 13 22:46:22.524613 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:46:22.524529 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hlll5\" (UniqueName: \"kubernetes.io/projected/9a4f8fb1-fb22-4a74-a2b9-e89938780f62-kube-api-access-hlll5\") pod \"maas-api-e2e-api-b-1276a7-5b54c55668-2lgc6\" (UID: \"9a4f8fb1-fb22-4a74-a2b9-e89938780f62\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-1276a7-5b54c55668-2lgc6" Jul 13 22:46:22.625849 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:46:22.625812 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9a4f8fb1-fb22-4a74-a2b9-e89938780f62-maas-api-tls\") pod \"maas-api-e2e-api-b-1276a7-5b54c55668-2lgc6\" (UID: \"9a4f8fb1-fb22-4a74-a2b9-e89938780f62\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-1276a7-5b54c55668-2lgc6" Jul 13 22:46:22.626016 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:46:22.625865 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9a4f8fb1-fb22-4a74-a2b9-e89938780f62-openshift-service-ca\") pod \"maas-api-e2e-api-b-1276a7-5b54c55668-2lgc6\" (UID: \"9a4f8fb1-fb22-4a74-a2b9-e89938780f62\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-1276a7-5b54c55668-2lgc6" Jul 13 22:46:22.626016 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:46:22.625930 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hlll5\" (UniqueName: \"kubernetes.io/projected/9a4f8fb1-fb22-4a74-a2b9-e89938780f62-kube-api-access-hlll5\") pod \"maas-api-e2e-api-b-1276a7-5b54c55668-2lgc6\" (UID: \"9a4f8fb1-fb22-4a74-a2b9-e89938780f62\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-1276a7-5b54c55668-2lgc6" Jul 13 22:46:22.626564 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:46:22.626541 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9a4f8fb1-fb22-4a74-a2b9-e89938780f62-openshift-service-ca\") pod \"maas-api-e2e-api-b-1276a7-5b54c55668-2lgc6\" (UID: \"9a4f8fb1-fb22-4a74-a2b9-e89938780f62\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-1276a7-5b54c55668-2lgc6" Jul 13 22:46:22.628297 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:46:22.628245 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9a4f8fb1-fb22-4a74-a2b9-e89938780f62-maas-api-tls\") pod \"maas-api-e2e-api-b-1276a7-5b54c55668-2lgc6\" (UID: \"9a4f8fb1-fb22-4a74-a2b9-e89938780f62\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-1276a7-5b54c55668-2lgc6" Jul 13 22:46:22.634806 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:46:22.634778 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hlll5\" (UniqueName: \"kubernetes.io/projected/9a4f8fb1-fb22-4a74-a2b9-e89938780f62-kube-api-access-hlll5\") pod \"maas-api-e2e-api-b-1276a7-5b54c55668-2lgc6\" (UID: \"9a4f8fb1-fb22-4a74-a2b9-e89938780f62\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-1276a7-5b54c55668-2lgc6" Jul 13 22:46:22.785692 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:46:22.785661 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-1276a7-5b54c55668-2lgc6" Jul 13 22:46:22.989356 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:46:22.989220 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-1276a7-5b54c55668-2lgc6"] Jul 13 22:46:22.991903 ip-10-0-133-179 kubenswrapper[2583]: W0713 22:46:22.991871 2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9a4f8fb1_fb22_4a74_a2b9_e89938780f62.slice/crio-f6d619fdd31fbd66230b327672be71984b3b7932433c6362fde7950b67d02b06 WatchSource:0}: Error finding container f6d619fdd31fbd66230b327672be71984b3b7932433c6362fde7950b67d02b06: Status 404 returned error can't find the container with id f6d619fdd31fbd66230b327672be71984b3b7932433c6362fde7950b67d02b06 Jul 13 22:46:23.396884 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:46:23.396849 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-1276a7-5b54c55668-2lgc6" event={"ID":"9a4f8fb1-fb22-4a74-a2b9-e89938780f62","Type":"ContainerStarted","Data":"f6d619fdd31fbd66230b327672be71984b3b7932433c6362fde7950b67d02b06"} Jul 13 22:46:24.402709 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:46:24.402669 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-1276a7-5b54c55668-2lgc6" event={"ID":"9a4f8fb1-fb22-4a74-a2b9-e89938780f62","Type":"ContainerStarted","Data":"36295e5fe57bef407363294a0d22e7de2db1162749ae8a10fa3aff6ff97d7cde"} Jul 13 22:46:24.403110 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:46:24.402851 2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-1276a7-5b54c55668-2lgc6" Jul 13 22:46:24.427797 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:46:24.427736 2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-1276a7-5b54c55668-2lgc6" podStartSLOduration=1.9546793409999998 podStartE2EDuration="2.42771892s" totalImagesPullingTime="473.039579ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-13 22:46:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 22:46:24.426654006 +0000 UTC m=+1979.674569358" watchObservedRunningTime="2026-07-13 22:46:24.42771892 +0000 UTC m=+1979.675634270" Jul 13 22:46:27.417451 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:46:27.417367 2583 generic.go:356] "Generic (PLEG): container finished" podID="950c3e6e-75a8-4e20-ba7f-84c0a08e1c1b" containerID="f4251a7a13c3037c580f09edee54ea25330c989784a5480395bd3cfa3b74ca51" exitCode=0 Jul 13 22:46:27.417859 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:46:27.417443 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-8f95ee-openshift-default-5df565bcf9-whs2f" event={"ID":"950c3e6e-75a8-4e20-ba7f-84c0a08e1c1b","Type":"ContainerDied","Data":"f4251a7a13c3037c580f09edee54ea25330c989784a5480395bd3cfa3b74ca51"} Jul 13 22:46:27.417859 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:46:27.417815 2583 scope.go:117] "RemoveContainer" containerID="f4251a7a13c3037c580f09edee54ea25330c989784a5480395bd3cfa3b74ca51" Jul 13 22:46:27.418916 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:46:27.418885 2583 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 13 22:46:27.418991 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:46:27.418974 2583 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 13 22:46:27.419038 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:46:27.419023 2583 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 13 22:46:28.347516 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:46:28.347465 2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-8f95ee-openshift-default-5df565bcf9-whs2f" Jul 13 22:46:28.424151 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:46:28.424118 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-8f95ee-openshift-default-5df565bcf9-whs2f" event={"ID":"950c3e6e-75a8-4e20-ba7f-84c0a08e1c1b","Type":"ContainerStarted","Data":"e511d8ba4aeb5794242f7fc183868f30b485f649f4f014ab18d8a69c3d2e5c60"} Jul 13 22:46:29.026424 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:46:29.026382 2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-8f95ee-openshift-default-5df565bcf9-whs2f" Jul 13 22:46:29.027552 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:46:29.027522 2583 patch_prober.go:28] interesting pod/e2e-api-a-8f95ee-openshift-default-5df565bcf9-whs2f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.37:15021/healthz/ready\": dial tcp 10.133.0.37:15021: connect: connection refused" start-of-body= Jul 13 22:46:29.027692 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:46:29.027600 2583 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-8f95ee-openshift-default-5df565bcf9-whs2f" podUID="950c3e6e-75a8-4e20-ba7f-84c0a08e1c1b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.37:15021/healthz/ready\": dial tcp 10.133.0.37:15021: connect: connection refused" Jul 13 22:46:30.027355 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:46:30.027314 2583 patch_prober.go:28] interesting pod/e2e-api-a-8f95ee-openshift-default-5df565bcf9-whs2f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.37:15021/healthz/ready\": dial tcp 10.133.0.37:15021: connect: connection refused" start-of-body= Jul 13 22:46:30.027789 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:46:30.027372 2583 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-8f95ee-openshift-default-5df565bcf9-whs2f" podUID="950c3e6e-75a8-4e20-ba7f-84c0a08e1c1b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.37:15021/healthz/ready\": dial tcp 10.133.0.37:15021: connect: connection refused" Jul 13 22:46:30.413223 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:46:30.413141 2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-1276a7-5b54c55668-2lgc6" Jul 13 22:46:31.027469 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:46:31.027422 2583 patch_prober.go:28] interesting pod/e2e-api-a-8f95ee-openshift-default-5df565bcf9-whs2f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.37:15021/healthz/ready\": dial tcp 10.133.0.37:15021: connect: connection refused" start-of-body= Jul 13 22:46:31.027887 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:46:31.027501 2583 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-8f95ee-openshift-default-5df565bcf9-whs2f" podUID="950c3e6e-75a8-4e20-ba7f-84c0a08e1c1b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.37:15021/healthz/ready\": dial tcp 10.133.0.37:15021: connect: connection refused" Jul 13 22:46:32.027362 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:46:32.027307 2583 patch_prober.go:28] interesting pod/e2e-api-a-8f95ee-openshift-default-5df565bcf9-whs2f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.37:15021/healthz/ready\": dial tcp 10.133.0.37:15021: connect: connection refused" start-of-body= Jul 13 22:46:32.027362 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:46:32.027370 2583 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-8f95ee-openshift-default-5df565bcf9-whs2f" podUID="950c3e6e-75a8-4e20-ba7f-84c0a08e1c1b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.37:15021/healthz/ready\": dial tcp 10.133.0.37:15021: connect: connection refused" Jul 13 22:46:33.037239 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:46:33.037206 2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-8f95ee-openshift-default-5df565bcf9-whs2f" Jul 13 22:46:33.038356 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:46:33.038329 2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-8f95ee-openshift-default-5df565bcf9-whs2f" Jul 13 22:46:33.042240 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:46:33.042222 2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-8f95ee-openshift-default-5df565bcf9-whs2f" Jul 13 22:46:35.600333 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:46:35.600241 2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-8f95ee-6779c6cb5c-rkjwz"] Jul 13 22:46:35.600873 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:46:35.600485 2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-8f95ee-6779c6cb5c-rkjwz" podUID="dadfd181-e4c0-4f74-841f-fc644964cbd1" containerName="maas-api" containerID="cri-o://50af117b02ca2c10c4098747ff3e31b7b8898994292a0eeb98e95b4ab82c96c6" gracePeriod=30 Jul 13 22:46:35.862620 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:46:35.862548 2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-8f95ee-6779c6cb5c-rkjwz" Jul 13 22:46:35.985192 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:46:35.985145 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/dadfd181-e4c0-4f74-841f-fc644964cbd1-openshift-service-ca\") pod \"dadfd181-e4c0-4f74-841f-fc644964cbd1\" (UID: \"dadfd181-e4c0-4f74-841f-fc644964cbd1\") " Jul 13 22:46:35.985398 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:46:35.985295 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/dadfd181-e4c0-4f74-841f-fc644964cbd1-maas-api-tls\") pod \"dadfd181-e4c0-4f74-841f-fc644964cbd1\" (UID: \"dadfd181-e4c0-4f74-841f-fc644964cbd1\") " Jul 13 22:46:35.985398 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:46:35.985337 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-n46vj\" (UniqueName: \"kubernetes.io/projected/dadfd181-e4c0-4f74-841f-fc644964cbd1-kube-api-access-n46vj\") pod \"dadfd181-e4c0-4f74-841f-fc644964cbd1\" (UID: \"dadfd181-e4c0-4f74-841f-fc644964cbd1\") " Jul 13 22:46:35.985553 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:46:35.985526 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dadfd181-e4c0-4f74-841f-fc644964cbd1-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "dadfd181-e4c0-4f74-841f-fc644964cbd1" (UID: "dadfd181-e4c0-4f74-841f-fc644964cbd1"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 13 22:46:35.987564 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:46:35.987535 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/dadfd181-e4c0-4f74-841f-fc644964cbd1-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "dadfd181-e4c0-4f74-841f-fc644964cbd1" (UID: "dadfd181-e4c0-4f74-841f-fc644964cbd1"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 13 22:46:35.987665 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:46:35.987644 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dadfd181-e4c0-4f74-841f-fc644964cbd1-kube-api-access-n46vj" (OuterVolumeSpecName: "kube-api-access-n46vj") pod "dadfd181-e4c0-4f74-841f-fc644964cbd1" (UID: "dadfd181-e4c0-4f74-841f-fc644964cbd1"). InnerVolumeSpecName "kube-api-access-n46vj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 13 22:46:36.087064 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:46:36.087022 2583 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/dadfd181-e4c0-4f74-841f-fc644964cbd1-openshift-service-ca\") on node \"ip-10-0-133-179.ec2.internal\" DevicePath \"\"" Jul 13 22:46:36.087064 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:46:36.087059 2583 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/dadfd181-e4c0-4f74-841f-fc644964cbd1-maas-api-tls\") on node \"ip-10-0-133-179.ec2.internal\" DevicePath \"\"" Jul 13 22:46:36.087064 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:46:36.087071 2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n46vj\" (UniqueName: \"kubernetes.io/projected/dadfd181-e4c0-4f74-841f-fc644964cbd1-kube-api-access-n46vj\") on node \"ip-10-0-133-179.ec2.internal\" DevicePath \"\"" Jul 13 22:46:36.467999 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:46:36.467958 2583 generic.go:356] "Generic (PLEG): container finished" podID="dadfd181-e4c0-4f74-841f-fc644964cbd1" containerID="50af117b02ca2c10c4098747ff3e31b7b8898994292a0eeb98e95b4ab82c96c6" exitCode=0 Jul 13 22:46:36.468170 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:46:36.468002 2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-8f95ee-6779c6cb5c-rkjwz" Jul 13 22:46:36.468170 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:46:36.468043 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-8f95ee-6779c6cb5c-rkjwz" event={"ID":"dadfd181-e4c0-4f74-841f-fc644964cbd1","Type":"ContainerDied","Data":"50af117b02ca2c10c4098747ff3e31b7b8898994292a0eeb98e95b4ab82c96c6"} Jul 13 22:46:36.468170 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:46:36.468101 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-8f95ee-6779c6cb5c-rkjwz" event={"ID":"dadfd181-e4c0-4f74-841f-fc644964cbd1","Type":"ContainerDied","Data":"09ae8980792ca7729910901678e150d02e2d8ff02a273752eae0fd1d34950120"} Jul 13 22:46:36.468170 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:46:36.468131 2583 scope.go:117] "RemoveContainer" containerID="50af117b02ca2c10c4098747ff3e31b7b8898994292a0eeb98e95b4ab82c96c6" Jul 13 22:46:36.480285 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:46:36.480257 2583 scope.go:117] "RemoveContainer" containerID="50af117b02ca2c10c4098747ff3e31b7b8898994292a0eeb98e95b4ab82c96c6" Jul 13 22:46:36.480656 ip-10-0-133-179 kubenswrapper[2583]: E0713 22:46:36.480633 2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"50af117b02ca2c10c4098747ff3e31b7b8898994292a0eeb98e95b4ab82c96c6\": container with ID starting with 50af117b02ca2c10c4098747ff3e31b7b8898994292a0eeb98e95b4ab82c96c6 not found: ID does not exist" containerID="50af117b02ca2c10c4098747ff3e31b7b8898994292a0eeb98e95b4ab82c96c6" Jul 13 22:46:36.480755 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:46:36.480669 2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"50af117b02ca2c10c4098747ff3e31b7b8898994292a0eeb98e95b4ab82c96c6"} err="failed to get container status \"50af117b02ca2c10c4098747ff3e31b7b8898994292a0eeb98e95b4ab82c96c6\": rpc error: code = NotFound desc = could not find container \"50af117b02ca2c10c4098747ff3e31b7b8898994292a0eeb98e95b4ab82c96c6\": container with ID starting with 50af117b02ca2c10c4098747ff3e31b7b8898994292a0eeb98e95b4ab82c96c6 not found: ID does not exist" Jul 13 22:46:36.490179 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:46:36.490155 2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-8f95ee-6779c6cb5c-rkjwz"] Jul 13 22:46:36.494297 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:46:36.494271 2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-8f95ee-6779c6cb5c-rkjwz"] Jul 13 22:46:37.390392 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:46:37.390356 2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="dadfd181-e4c0-4f74-841f-fc644964cbd1" path="/var/lib/kubelet/pods/dadfd181-e4c0-4f74-841f-fc644964cbd1/volumes" Jul 13 22:46:38.063305 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:46:38.062949 2583 patch_prober.go:28] interesting pod/e2e-api-b-1276a7-openshift-default-84f5889fb9-2qzdr container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.39:15021/healthz/ready\": read tcp 10.133.0.2:48362->10.133.0.39:15021: read: connection reset by peer" start-of-body= Jul 13 22:46:38.063305 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:46:38.063010 2583 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-b-1276a7-openshift-default-84f5889fb9-2qzdr" podUID="0b5a60d1-133b-4e0b-b868-fbd4adc706db" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.39:15021/healthz/ready\": read tcp 10.133.0.2:48362->10.133.0.39:15021: read: connection reset by peer" Jul 13 22:46:38.479990 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:46:38.479897 2583 generic.go:356] "Generic (PLEG): container finished" podID="0b5a60d1-133b-4e0b-b868-fbd4adc706db" containerID="b98188f01ac9ee00971d7d1060c4934b04e42a7424cc8cf322450e6cf55c4dff" exitCode=0 Jul 13 22:46:38.479990 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:46:38.479972 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-1276a7-openshift-default-84f5889fb9-2qzdr" event={"ID":"0b5a60d1-133b-4e0b-b868-fbd4adc706db","Type":"ContainerDied","Data":"b98188f01ac9ee00971d7d1060c4934b04e42a7424cc8cf322450e6cf55c4dff"} Jul 13 22:46:38.480453 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:46:38.480374 2583 scope.go:117] "RemoveContainer" containerID="b98188f01ac9ee00971d7d1060c4934b04e42a7424cc8cf322450e6cf55c4dff" Jul 13 22:46:38.481346 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:46:38.481317 2583 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 13 22:46:38.481440 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:46:38.481403 2583 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 13 22:46:38.481505 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:46:38.481451 2583 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 13 22:46:39.493118 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:46:39.493082 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-1276a7-openshift-default-84f5889fb9-2qzdr" event={"ID":"0b5a60d1-133b-4e0b-b868-fbd4adc706db","Type":"ContainerStarted","Data":"14bb5b2bf3bbf2709f7a53e1378c91164d73b284a5a825aad2f078daa401e66d"} Jul 13 22:46:39.814548 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:46:39.814503 2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-1276a7-openshift-default-84f5889fb9-2qzdr" Jul 13 22:46:39.814844 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:46:39.814801 2583 patch_prober.go:28] interesting pod/e2e-api-b-1276a7-openshift-default-84f5889fb9-2qzdr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.39:15021/healthz/ready\": dial tcp 10.133.0.39:15021: connect: connection refused" start-of-body= Jul 13 22:46:39.814892 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:46:39.814853 2583 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-1276a7-openshift-default-84f5889fb9-2qzdr" podUID="0b5a60d1-133b-4e0b-b868-fbd4adc706db" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.39:15021/healthz/ready\": dial tcp 10.133.0.39:15021: connect: connection refused" Jul 13 22:46:40.815103 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:46:40.815058 2583 patch_prober.go:28] interesting pod/e2e-api-b-1276a7-openshift-default-84f5889fb9-2qzdr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.39:15021/healthz/ready\": dial tcp 10.133.0.39:15021: connect: connection refused" start-of-body= Jul 13 22:46:40.815499 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:46:40.815125 2583 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-1276a7-openshift-default-84f5889fb9-2qzdr" podUID="0b5a60d1-133b-4e0b-b868-fbd4adc706db" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.39:15021/healthz/ready\": dial tcp 10.133.0.39:15021: connect: connection refused" Jul 13 22:46:41.646301 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:46:41.646243 2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-8f95ee-openshift-default-5df565bcf9-whs2f"] Jul 13 22:46:41.659483 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:46:41.659427 2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-api-a-8f95ee-openshift-default-5df565bcf9-whs2f" podUID="950c3e6e-75a8-4e20-ba7f-84c0a08e1c1b" containerName="istio-proxy" containerID="cri-o://e511d8ba4aeb5794242f7fc183868f30b485f649f4f014ab18d8a69c3d2e5c60" gracePeriod=30 Jul 13 22:46:41.814494 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:46:41.814458 2583 patch_prober.go:28] interesting pod/e2e-api-b-1276a7-openshift-default-84f5889fb9-2qzdr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.39:15021/healthz/ready\": dial tcp 10.133.0.39:15021: connect: connection refused" start-of-body= Jul 13 22:46:41.814702 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:46:41.814515 2583 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-1276a7-openshift-default-84f5889fb9-2qzdr" podUID="0b5a60d1-133b-4e0b-b868-fbd4adc706db" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.39:15021/healthz/ready\": dial tcp 10.133.0.39:15021: connect: connection refused" Jul 13 22:46:42.814670 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:46:42.814630 2583 patch_prober.go:28] interesting pod/e2e-api-b-1276a7-openshift-default-84f5889fb9-2qzdr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.39:15021/healthz/ready\": dial tcp 10.133.0.39:15021: connect: connection refused" start-of-body= Jul 13 22:46:42.815132 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:46:42.814692 2583 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-1276a7-openshift-default-84f5889fb9-2qzdr" podUID="0b5a60d1-133b-4e0b-b868-fbd4adc706db" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.39:15021/healthz/ready\": dial tcp 10.133.0.39:15021: connect: connection refused" Jul 13 22:46:42.827993 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:46:42.827959 2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-1276a7-5b54c55668-2lgc6"] Jul 13 22:46:42.828221 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:46:42.828195 2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-1276a7-5b54c55668-2lgc6" podUID="9a4f8fb1-fb22-4a74-a2b9-e89938780f62" containerName="maas-api" containerID="cri-o://36295e5fe57bef407363294a0d22e7de2db1162749ae8a10fa3aff6ff97d7cde" gracePeriod=30 Jul 13 22:46:43.106403 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:46:43.106377 2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-1276a7-5b54c55668-2lgc6" Jul 13 22:46:43.153621 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:46:43.152838 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9a4f8fb1-fb22-4a74-a2b9-e89938780f62-openshift-service-ca\") pod \"9a4f8fb1-fb22-4a74-a2b9-e89938780f62\" (UID: \"9a4f8fb1-fb22-4a74-a2b9-e89938780f62\") " Jul 13 22:46:43.153621 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:46:43.152888 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hlll5\" (UniqueName: \"kubernetes.io/projected/9a4f8fb1-fb22-4a74-a2b9-e89938780f62-kube-api-access-hlll5\") pod \"9a4f8fb1-fb22-4a74-a2b9-e89938780f62\" (UID: \"9a4f8fb1-fb22-4a74-a2b9-e89938780f62\") " Jul 13 22:46:43.153621 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:46:43.153013 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9a4f8fb1-fb22-4a74-a2b9-e89938780f62-maas-api-tls\") pod \"9a4f8fb1-fb22-4a74-a2b9-e89938780f62\" (UID: \"9a4f8fb1-fb22-4a74-a2b9-e89938780f62\") " Jul 13 22:46:43.154592 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:46:43.153950 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9a4f8fb1-fb22-4a74-a2b9-e89938780f62-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "9a4f8fb1-fb22-4a74-a2b9-e89938780f62" (UID: "9a4f8fb1-fb22-4a74-a2b9-e89938780f62"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 13 22:46:43.157080 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:46:43.157042 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9a4f8fb1-fb22-4a74-a2b9-e89938780f62-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "9a4f8fb1-fb22-4a74-a2b9-e89938780f62" (UID: "9a4f8fb1-fb22-4a74-a2b9-e89938780f62"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 13 22:46:43.158034 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:46:43.157998 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9a4f8fb1-fb22-4a74-a2b9-e89938780f62-kube-api-access-hlll5" (OuterVolumeSpecName: "kube-api-access-hlll5") pod "9a4f8fb1-fb22-4a74-a2b9-e89938780f62" (UID: "9a4f8fb1-fb22-4a74-a2b9-e89938780f62"). InnerVolumeSpecName "kube-api-access-hlll5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 13 22:46:43.254139 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:46:43.254085 2583 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9a4f8fb1-fb22-4a74-a2b9-e89938780f62-maas-api-tls\") on node \"ip-10-0-133-179.ec2.internal\" DevicePath \"\"" Jul 13 22:46:43.254139 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:46:43.254140 2583 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9a4f8fb1-fb22-4a74-a2b9-e89938780f62-openshift-service-ca\") on node \"ip-10-0-133-179.ec2.internal\" DevicePath \"\"" Jul 13 22:46:43.254357 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:46:43.254156 2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hlll5\" (UniqueName: \"kubernetes.io/projected/9a4f8fb1-fb22-4a74-a2b9-e89938780f62-kube-api-access-hlll5\") on node \"ip-10-0-133-179.ec2.internal\" DevicePath \"\"" Jul 13 22:46:43.510865 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:46:43.510824 2583 generic.go:356] "Generic (PLEG): container finished" podID="9a4f8fb1-fb22-4a74-a2b9-e89938780f62" containerID="36295e5fe57bef407363294a0d22e7de2db1162749ae8a10fa3aff6ff97d7cde" exitCode=0 Jul 13 22:46:43.510865 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:46:43.510866 2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-1276a7-5b54c55668-2lgc6" Jul 13 22:46:43.511124 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:46:43.510905 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-1276a7-5b54c55668-2lgc6" event={"ID":"9a4f8fb1-fb22-4a74-a2b9-e89938780f62","Type":"ContainerDied","Data":"36295e5fe57bef407363294a0d22e7de2db1162749ae8a10fa3aff6ff97d7cde"} Jul 13 22:46:43.511124 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:46:43.510964 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-1276a7-5b54c55668-2lgc6" event={"ID":"9a4f8fb1-fb22-4a74-a2b9-e89938780f62","Type":"ContainerDied","Data":"f6d619fdd31fbd66230b327672be71984b3b7932433c6362fde7950b67d02b06"} Jul 13 22:46:43.511124 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:46:43.510996 2583 scope.go:117] "RemoveContainer" containerID="36295e5fe57bef407363294a0d22e7de2db1162749ae8a10fa3aff6ff97d7cde" Jul 13 22:46:43.520553 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:46:43.520534 2583 scope.go:117] "RemoveContainer" containerID="36295e5fe57bef407363294a0d22e7de2db1162749ae8a10fa3aff6ff97d7cde" Jul 13 22:46:43.520852 ip-10-0-133-179 kubenswrapper[2583]: E0713 22:46:43.520832 2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"36295e5fe57bef407363294a0d22e7de2db1162749ae8a10fa3aff6ff97d7cde\": container with ID starting with 36295e5fe57bef407363294a0d22e7de2db1162749ae8a10fa3aff6ff97d7cde not found: ID does not exist" containerID="36295e5fe57bef407363294a0d22e7de2db1162749ae8a10fa3aff6ff97d7cde" Jul 13 22:46:43.520906 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:46:43.520861 2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"36295e5fe57bef407363294a0d22e7de2db1162749ae8a10fa3aff6ff97d7cde"} err="failed to get container status \"36295e5fe57bef407363294a0d22e7de2db1162749ae8a10fa3aff6ff97d7cde\": rpc error: code = NotFound desc = could not find container \"36295e5fe57bef407363294a0d22e7de2db1162749ae8a10fa3aff6ff97d7cde\": container with ID starting with 36295e5fe57bef407363294a0d22e7de2db1162749ae8a10fa3aff6ff97d7cde not found: ID does not exist" Jul 13 22:46:43.526213 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:46:43.526188 2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-1276a7-5b54c55668-2lgc6"] Jul 13 22:46:43.529342 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:46:43.529317 2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-1276a7-5b54c55668-2lgc6"] Jul 13 22:46:43.817544 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:46:43.817513 2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-1276a7-openshift-default-84f5889fb9-2qzdr" Jul 13 22:46:43.817923 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:46:43.817789 2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-1276a7-openshift-default-84f5889fb9-2qzdr" Jul 13 22:46:43.818589 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:46:43.818556 2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-1276a7-openshift-default-84f5889fb9-2qzdr" Jul 13 22:46:45.389000 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:46:45.388964 2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9a4f8fb1-fb22-4a74-a2b9-e89938780f62" path="/var/lib/kubelet/pods/9a4f8fb1-fb22-4a74-a2b9-e89938780f62/volumes" Jul 13 22:46:46.915012 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:46:46.914986 2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-8f95ee-openshift-default-5df565bcf9-whs2f" Jul 13 22:46:46.990282 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:46:46.990180 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/950c3e6e-75a8-4e20-ba7f-84c0a08e1c1b-istio-podinfo\") pod \"950c3e6e-75a8-4e20-ba7f-84c0a08e1c1b\" (UID: \"950c3e6e-75a8-4e20-ba7f-84c0a08e1c1b\") " Jul 13 22:46:46.990282 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:46:46.990221 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/950c3e6e-75a8-4e20-ba7f-84c0a08e1c1b-istio-data\") pod \"950c3e6e-75a8-4e20-ba7f-84c0a08e1c1b\" (UID: \"950c3e6e-75a8-4e20-ba7f-84c0a08e1c1b\") " Jul 13 22:46:46.990282 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:46:46.990246 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/950c3e6e-75a8-4e20-ba7f-84c0a08e1c1b-workload-certs\") pod \"950c3e6e-75a8-4e20-ba7f-84c0a08e1c1b\" (UID: \"950c3e6e-75a8-4e20-ba7f-84c0a08e1c1b\") " Jul 13 22:46:46.990562 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:46:46.990287 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2f854\" (UniqueName: \"kubernetes.io/projected/950c3e6e-75a8-4e20-ba7f-84c0a08e1c1b-kube-api-access-2f854\") pod \"950c3e6e-75a8-4e20-ba7f-84c0a08e1c1b\" (UID: \"950c3e6e-75a8-4e20-ba7f-84c0a08e1c1b\") " Jul 13 22:46:46.990562 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:46:46.990313 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/950c3e6e-75a8-4e20-ba7f-84c0a08e1c1b-istio-envoy\") pod \"950c3e6e-75a8-4e20-ba7f-84c0a08e1c1b\" (UID: \"950c3e6e-75a8-4e20-ba7f-84c0a08e1c1b\") " Jul 13 22:46:46.990562 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:46:46.990336 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/950c3e6e-75a8-4e20-ba7f-84c0a08e1c1b-istio-token\") pod \"950c3e6e-75a8-4e20-ba7f-84c0a08e1c1b\" (UID: \"950c3e6e-75a8-4e20-ba7f-84c0a08e1c1b\") " Jul 13 22:46:46.990562 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:46:46.990378 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/950c3e6e-75a8-4e20-ba7f-84c0a08e1c1b-workload-socket\") pod \"950c3e6e-75a8-4e20-ba7f-84c0a08e1c1b\" (UID: \"950c3e6e-75a8-4e20-ba7f-84c0a08e1c1b\") " Jul 13 22:46:46.990562 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:46:46.990424 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/950c3e6e-75a8-4e20-ba7f-84c0a08e1c1b-istiod-ca-cert\") pod \"950c3e6e-75a8-4e20-ba7f-84c0a08e1c1b\" (UID: \"950c3e6e-75a8-4e20-ba7f-84c0a08e1c1b\") " Jul 13 22:46:46.990562 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:46:46.990457 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/950c3e6e-75a8-4e20-ba7f-84c0a08e1c1b-credential-socket\") pod \"950c3e6e-75a8-4e20-ba7f-84c0a08e1c1b\" (UID: \"950c3e6e-75a8-4e20-ba7f-84c0a08e1c1b\") " Jul 13 22:46:46.990942 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:46:46.990653 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/950c3e6e-75a8-4e20-ba7f-84c0a08e1c1b-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "950c3e6e-75a8-4e20-ba7f-84c0a08e1c1b" (UID: "950c3e6e-75a8-4e20-ba7f-84c0a08e1c1b"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 13 22:46:46.990942 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:46:46.990715 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/950c3e6e-75a8-4e20-ba7f-84c0a08e1c1b-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "950c3e6e-75a8-4e20-ba7f-84c0a08e1c1b" (UID: "950c3e6e-75a8-4e20-ba7f-84c0a08e1c1b"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 13 22:46:46.990942 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:46:46.990726 2583 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/950c3e6e-75a8-4e20-ba7f-84c0a08e1c1b-workload-certs\") on node \"ip-10-0-133-179.ec2.internal\" DevicePath \"\"" Jul 13 22:46:46.990942 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:46:46.990853 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/950c3e6e-75a8-4e20-ba7f-84c0a08e1c1b-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "950c3e6e-75a8-4e20-ba7f-84c0a08e1c1b" (UID: "950c3e6e-75a8-4e20-ba7f-84c0a08e1c1b"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 13 22:46:46.991147 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:46:46.990944 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/950c3e6e-75a8-4e20-ba7f-84c0a08e1c1b-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "950c3e6e-75a8-4e20-ba7f-84c0a08e1c1b" (UID: "950c3e6e-75a8-4e20-ba7f-84c0a08e1c1b"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 13 22:46:46.991147 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:46:46.991120 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/950c3e6e-75a8-4e20-ba7f-84c0a08e1c1b-istio-data" (OuterVolumeSpecName: "istio-data") pod "950c3e6e-75a8-4e20-ba7f-84c0a08e1c1b" (UID: "950c3e6e-75a8-4e20-ba7f-84c0a08e1c1b"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 13 22:46:46.992719 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:46:46.992691 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/950c3e6e-75a8-4e20-ba7f-84c0a08e1c1b-kube-api-access-2f854" (OuterVolumeSpecName: "kube-api-access-2f854") pod "950c3e6e-75a8-4e20-ba7f-84c0a08e1c1b" (UID: "950c3e6e-75a8-4e20-ba7f-84c0a08e1c1b"). InnerVolumeSpecName "kube-api-access-2f854". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 13 22:46:46.992850 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:46:46.992712 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/950c3e6e-75a8-4e20-ba7f-84c0a08e1c1b-istio-token" (OuterVolumeSpecName: "istio-token") pod "950c3e6e-75a8-4e20-ba7f-84c0a08e1c1b" (UID: "950c3e6e-75a8-4e20-ba7f-84c0a08e1c1b"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 13 22:46:46.992991 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:46:46.992974 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/950c3e6e-75a8-4e20-ba7f-84c0a08e1c1b-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "950c3e6e-75a8-4e20-ba7f-84c0a08e1c1b" (UID: "950c3e6e-75a8-4e20-ba7f-84c0a08e1c1b"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 13 22:46:46.993033 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:46:46.992990 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/950c3e6e-75a8-4e20-ba7f-84c0a08e1c1b-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "950c3e6e-75a8-4e20-ba7f-84c0a08e1c1b" (UID: "950c3e6e-75a8-4e20-ba7f-84c0a08e1c1b"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 13 22:46:47.091532 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:46:47.091477 2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2f854\" (UniqueName: \"kubernetes.io/projected/950c3e6e-75a8-4e20-ba7f-84c0a08e1c1b-kube-api-access-2f854\") on node \"ip-10-0-133-179.ec2.internal\" DevicePath \"\"" Jul 13 22:46:47.091532 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:46:47.091523 2583 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/950c3e6e-75a8-4e20-ba7f-84c0a08e1c1b-istio-envoy\") on node \"ip-10-0-133-179.ec2.internal\" DevicePath \"\"" Jul 13 22:46:47.091532 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:46:47.091533 2583 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/950c3e6e-75a8-4e20-ba7f-84c0a08e1c1b-istio-token\") on node \"ip-10-0-133-179.ec2.internal\" DevicePath \"\"" Jul 13 22:46:47.091532 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:46:47.091542 2583 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/950c3e6e-75a8-4e20-ba7f-84c0a08e1c1b-workload-socket\") on node \"ip-10-0-133-179.ec2.internal\" DevicePath \"\"" Jul 13 22:46:47.091532 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:46:47.091550 2583 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/950c3e6e-75a8-4e20-ba7f-84c0a08e1c1b-istiod-ca-cert\") on node \"ip-10-0-133-179.ec2.internal\" DevicePath \"\"" Jul 13 22:46:47.091867 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:46:47.091558 2583 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/950c3e6e-75a8-4e20-ba7f-84c0a08e1c1b-credential-socket\") on node \"ip-10-0-133-179.ec2.internal\" DevicePath \"\"" Jul 13 22:46:47.091867 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:46:47.091594 2583 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/950c3e6e-75a8-4e20-ba7f-84c0a08e1c1b-istio-podinfo\") on node \"ip-10-0-133-179.ec2.internal\" DevicePath \"\"" Jul 13 22:46:47.091867 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:46:47.091603 2583 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/950c3e6e-75a8-4e20-ba7f-84c0a08e1c1b-istio-data\") on node \"ip-10-0-133-179.ec2.internal\" DevicePath \"\"" Jul 13 22:46:47.531616 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:46:47.531556 2583 generic.go:356] "Generic (PLEG): container finished" podID="950c3e6e-75a8-4e20-ba7f-84c0a08e1c1b" containerID="e511d8ba4aeb5794242f7fc183868f30b485f649f4f014ab18d8a69c3d2e5c60" exitCode=0 Jul 13 22:46:47.531616 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:46:47.531601 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-8f95ee-openshift-default-5df565bcf9-whs2f" event={"ID":"950c3e6e-75a8-4e20-ba7f-84c0a08e1c1b","Type":"ContainerDied","Data":"e511d8ba4aeb5794242f7fc183868f30b485f649f4f014ab18d8a69c3d2e5c60"} Jul 13 22:46:47.531868 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:46:47.531628 2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-8f95ee-openshift-default-5df565bcf9-whs2f" Jul 13 22:46:47.531868 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:46:47.531683 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-8f95ee-openshift-default-5df565bcf9-whs2f" event={"ID":"950c3e6e-75a8-4e20-ba7f-84c0a08e1c1b","Type":"ContainerDied","Data":"236baadd5c7958f0da41a7cadb92644c89352c3da2e2eb1739131fa1e29d241e"} Jul 13 22:46:47.531868 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:46:47.531714 2583 scope.go:117] "RemoveContainer" containerID="e511d8ba4aeb5794242f7fc183868f30b485f649f4f014ab18d8a69c3d2e5c60" Jul 13 22:46:47.540127 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:46:47.540101 2583 scope.go:117] "RemoveContainer" containerID="f4251a7a13c3037c580f09edee54ea25330c989784a5480395bd3cfa3b74ca51" Jul 13 22:46:47.547094 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:46:47.547072 2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-8f95ee-openshift-default-5df565bcf9-whs2f"] Jul 13 22:46:47.547789 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:46:47.547768 2583 scope.go:117] "RemoveContainer" containerID="e511d8ba4aeb5794242f7fc183868f30b485f649f4f014ab18d8a69c3d2e5c60" Jul 13 22:46:47.548039 ip-10-0-133-179 kubenswrapper[2583]: E0713 22:46:47.548020 2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e511d8ba4aeb5794242f7fc183868f30b485f649f4f014ab18d8a69c3d2e5c60\": container with ID starting with e511d8ba4aeb5794242f7fc183868f30b485f649f4f014ab18d8a69c3d2e5c60 not found: ID does not exist" containerID="e511d8ba4aeb5794242f7fc183868f30b485f649f4f014ab18d8a69c3d2e5c60" Jul 13 22:46:47.548118 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:46:47.548053 2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e511d8ba4aeb5794242f7fc183868f30b485f649f4f014ab18d8a69c3d2e5c60"} err="failed to get container status \"e511d8ba4aeb5794242f7fc183868f30b485f649f4f014ab18d8a69c3d2e5c60\": rpc error: code = NotFound desc = could not find container \"e511d8ba4aeb5794242f7fc183868f30b485f649f4f014ab18d8a69c3d2e5c60\": container with ID starting with e511d8ba4aeb5794242f7fc183868f30b485f649f4f014ab18d8a69c3d2e5c60 not found: ID does not exist" Jul 13 22:46:47.548118 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:46:47.548078 2583 scope.go:117] "RemoveContainer" containerID="f4251a7a13c3037c580f09edee54ea25330c989784a5480395bd3cfa3b74ca51" Jul 13 22:46:47.548365 ip-10-0-133-179 kubenswrapper[2583]: E0713 22:46:47.548347 2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f4251a7a13c3037c580f09edee54ea25330c989784a5480395bd3cfa3b74ca51\": container with ID starting with f4251a7a13c3037c580f09edee54ea25330c989784a5480395bd3cfa3b74ca51 not found: ID does not exist" containerID="f4251a7a13c3037c580f09edee54ea25330c989784a5480395bd3cfa3b74ca51" Jul 13 22:46:47.548409 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:46:47.548370 2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f4251a7a13c3037c580f09edee54ea25330c989784a5480395bd3cfa3b74ca51"} err="failed to get container status \"f4251a7a13c3037c580f09edee54ea25330c989784a5480395bd3cfa3b74ca51\": rpc error: code = NotFound desc = could not find container \"f4251a7a13c3037c580f09edee54ea25330c989784a5480395bd3cfa3b74ca51\": container with ID starting with f4251a7a13c3037c580f09edee54ea25330c989784a5480395bd3cfa3b74ca51 not found: ID does not exist" Jul 13 22:46:47.550947 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:46:47.550926 2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-a-8f95ee-openshift-default-5df565bcf9-whs2f"] Jul 13 22:46:48.608515 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:46:48.608475 2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-1276a7-openshift-default-84f5889fb9-2qzdr"] Jul 13 22:46:48.608922 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:46:48.608838 2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-api-b-1276a7-openshift-default-84f5889fb9-2qzdr" podUID="0b5a60d1-133b-4e0b-b868-fbd4adc706db" containerName="istio-proxy" containerID="cri-o://14bb5b2bf3bbf2709f7a53e1378c91164d73b284a5a825aad2f078daa401e66d" gracePeriod=30 Jul 13 22:46:49.389332 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:46:49.389295 2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="950c3e6e-75a8-4e20-ba7f-84c0a08e1c1b" path="/var/lib/kubelet/pods/950c3e6e-75a8-4e20-ba7f-84c0a08e1c1b/volumes" Jul 13 22:46:49.493602 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:46:49.493556 2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-455d3b-openshift-default-7b9767fd8c-wtg94"] Jul 13 22:46:49.494046 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:46:49.494030 2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="950c3e6e-75a8-4e20-ba7f-84c0a08e1c1b" containerName="istio-proxy" Jul 13 22:46:49.494154 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:46:49.494048 2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="950c3e6e-75a8-4e20-ba7f-84c0a08e1c1b" containerName="istio-proxy" Jul 13 22:46:49.494154 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:46:49.494073 2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="dadfd181-e4c0-4f74-841f-fc644964cbd1" containerName="maas-api" Jul 13 22:46:49.494154 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:46:49.494110 2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="dadfd181-e4c0-4f74-841f-fc644964cbd1" containerName="maas-api" Jul 13 22:46:49.494154 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:46:49.494122 2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9a4f8fb1-fb22-4a74-a2b9-e89938780f62" containerName="maas-api" Jul 13 22:46:49.494154 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:46:49.494130 2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="9a4f8fb1-fb22-4a74-a2b9-e89938780f62" containerName="maas-api" Jul 13 22:46:49.494394 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:46:49.494228 2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="950c3e6e-75a8-4e20-ba7f-84c0a08e1c1b" containerName="istio-proxy" Jul 13 22:46:49.494394 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:46:49.494241 2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="dadfd181-e4c0-4f74-841f-fc644964cbd1" containerName="maas-api" Jul 13 22:46:49.494394 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:46:49.494249 2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="950c3e6e-75a8-4e20-ba7f-84c0a08e1c1b" containerName="istio-proxy" Jul 13 22:46:49.494394 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:46:49.494259 2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="9a4f8fb1-fb22-4a74-a2b9-e89938780f62" containerName="maas-api" Jul 13 22:46:49.497374 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:46:49.497354 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-455d3b-openshift-default-7b9767fd8c-wtg94" Jul 13 22:46:49.499555 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:46:49.499536 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-455d3b-openshift-default-dockercfg-k4nbx\"" Jul 13 22:46:49.509372 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:46:49.509351 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-455d3b-openshift-default-7b9767fd8c-wtg94"] Jul 13 22:46:49.614202 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:46:49.614166 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3b6b3625-bfa7-4be3-9b53-15294800bd4b-credential-socket\") pod \"e2e-shared-a-455d3b-openshift-default-7b9767fd8c-wtg94\" (UID: \"3b6b3625-bfa7-4be3-9b53-15294800bd4b\") " pod="openshift-ingress/e2e-shared-a-455d3b-openshift-default-7b9767fd8c-wtg94" Jul 13 22:46:49.614603 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:46:49.614212 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3b6b3625-bfa7-4be3-9b53-15294800bd4b-istio-podinfo\") pod \"e2e-shared-a-455d3b-openshift-default-7b9767fd8c-wtg94\" (UID: \"3b6b3625-bfa7-4be3-9b53-15294800bd4b\") " pod="openshift-ingress/e2e-shared-a-455d3b-openshift-default-7b9767fd8c-wtg94" Jul 13 22:46:49.614603 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:46:49.614240 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3b6b3625-bfa7-4be3-9b53-15294800bd4b-istio-envoy\") pod \"e2e-shared-a-455d3b-openshift-default-7b9767fd8c-wtg94\" (UID: \"3b6b3625-bfa7-4be3-9b53-15294800bd4b\") " pod="openshift-ingress/e2e-shared-a-455d3b-openshift-default-7b9767fd8c-wtg94" Jul 13 22:46:49.614603 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:46:49.614300 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3b6b3625-bfa7-4be3-9b53-15294800bd4b-istio-data\") pod \"e2e-shared-a-455d3b-openshift-default-7b9767fd8c-wtg94\" (UID: \"3b6b3625-bfa7-4be3-9b53-15294800bd4b\") " pod="openshift-ingress/e2e-shared-a-455d3b-openshift-default-7b9767fd8c-wtg94" Jul 13 22:46:49.614603 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:46:49.614374 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3b6b3625-bfa7-4be3-9b53-15294800bd4b-workload-socket\") pod \"e2e-shared-a-455d3b-openshift-default-7b9767fd8c-wtg94\" (UID: \"3b6b3625-bfa7-4be3-9b53-15294800bd4b\") " pod="openshift-ingress/e2e-shared-a-455d3b-openshift-default-7b9767fd8c-wtg94" Jul 13 22:46:49.614603 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:46:49.614434 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3b6b3625-bfa7-4be3-9b53-15294800bd4b-istiod-ca-cert\") pod \"e2e-shared-a-455d3b-openshift-default-7b9767fd8c-wtg94\" (UID: \"3b6b3625-bfa7-4be3-9b53-15294800bd4b\") " pod="openshift-ingress/e2e-shared-a-455d3b-openshift-default-7b9767fd8c-wtg94" Jul 13 22:46:49.614603 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:46:49.614465 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5t626\" (UniqueName: \"kubernetes.io/projected/3b6b3625-bfa7-4be3-9b53-15294800bd4b-kube-api-access-5t626\") pod \"e2e-shared-a-455d3b-openshift-default-7b9767fd8c-wtg94\" (UID: \"3b6b3625-bfa7-4be3-9b53-15294800bd4b\") " pod="openshift-ingress/e2e-shared-a-455d3b-openshift-default-7b9767fd8c-wtg94" Jul 13 22:46:49.614603 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:46:49.614530 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3b6b3625-bfa7-4be3-9b53-15294800bd4b-istio-token\") pod \"e2e-shared-a-455d3b-openshift-default-7b9767fd8c-wtg94\" (UID: \"3b6b3625-bfa7-4be3-9b53-15294800bd4b\") " pod="openshift-ingress/e2e-shared-a-455d3b-openshift-default-7b9767fd8c-wtg94" Jul 13 22:46:49.614603 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:46:49.614600 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3b6b3625-bfa7-4be3-9b53-15294800bd4b-workload-certs\") pod \"e2e-shared-a-455d3b-openshift-default-7b9767fd8c-wtg94\" (UID: \"3b6b3625-bfa7-4be3-9b53-15294800bd4b\") " pod="openshift-ingress/e2e-shared-a-455d3b-openshift-default-7b9767fd8c-wtg94" Jul 13 22:46:49.715449 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:46:49.715359 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3b6b3625-bfa7-4be3-9b53-15294800bd4b-istio-podinfo\") pod \"e2e-shared-a-455d3b-openshift-default-7b9767fd8c-wtg94\" (UID: \"3b6b3625-bfa7-4be3-9b53-15294800bd4b\") " pod="openshift-ingress/e2e-shared-a-455d3b-openshift-default-7b9767fd8c-wtg94" Jul 13 22:46:49.715449 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:46:49.715405 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3b6b3625-bfa7-4be3-9b53-15294800bd4b-istio-envoy\") pod \"e2e-shared-a-455d3b-openshift-default-7b9767fd8c-wtg94\" (UID: \"3b6b3625-bfa7-4be3-9b53-15294800bd4b\") " pod="openshift-ingress/e2e-shared-a-455d3b-openshift-default-7b9767fd8c-wtg94" Jul 13 22:46:49.715449 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:46:49.715429 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3b6b3625-bfa7-4be3-9b53-15294800bd4b-istio-data\") pod \"e2e-shared-a-455d3b-openshift-default-7b9767fd8c-wtg94\" (UID: \"3b6b3625-bfa7-4be3-9b53-15294800bd4b\") " pod="openshift-ingress/e2e-shared-a-455d3b-openshift-default-7b9767fd8c-wtg94" Jul 13 22:46:49.715774 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:46:49.715457 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3b6b3625-bfa7-4be3-9b53-15294800bd4b-workload-socket\") pod \"e2e-shared-a-455d3b-openshift-default-7b9767fd8c-wtg94\" (UID: \"3b6b3625-bfa7-4be3-9b53-15294800bd4b\") " pod="openshift-ingress/e2e-shared-a-455d3b-openshift-default-7b9767fd8c-wtg94" Jul 13 22:46:49.715774 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:46:49.715481 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3b6b3625-bfa7-4be3-9b53-15294800bd4b-istiod-ca-cert\") pod \"e2e-shared-a-455d3b-openshift-default-7b9767fd8c-wtg94\" (UID: \"3b6b3625-bfa7-4be3-9b53-15294800bd4b\") " pod="openshift-ingress/e2e-shared-a-455d3b-openshift-default-7b9767fd8c-wtg94" Jul 13 22:46:49.715774 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:46:49.715498 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5t626\" (UniqueName: \"kubernetes.io/projected/3b6b3625-bfa7-4be3-9b53-15294800bd4b-kube-api-access-5t626\") pod \"e2e-shared-a-455d3b-openshift-default-7b9767fd8c-wtg94\" (UID: \"3b6b3625-bfa7-4be3-9b53-15294800bd4b\") " pod="openshift-ingress/e2e-shared-a-455d3b-openshift-default-7b9767fd8c-wtg94" Jul 13 22:46:49.715774 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:46:49.715691 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3b6b3625-bfa7-4be3-9b53-15294800bd4b-istio-token\") pod \"e2e-shared-a-455d3b-openshift-default-7b9767fd8c-wtg94\" (UID: \"3b6b3625-bfa7-4be3-9b53-15294800bd4b\") " pod="openshift-ingress/e2e-shared-a-455d3b-openshift-default-7b9767fd8c-wtg94" Jul 13 22:46:49.715774 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:46:49.715770 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3b6b3625-bfa7-4be3-9b53-15294800bd4b-workload-certs\") pod \"e2e-shared-a-455d3b-openshift-default-7b9767fd8c-wtg94\" (UID: \"3b6b3625-bfa7-4be3-9b53-15294800bd4b\") " pod="openshift-ingress/e2e-shared-a-455d3b-openshift-default-7b9767fd8c-wtg94" Jul 13 22:46:49.716063 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:46:49.715822 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3b6b3625-bfa7-4be3-9b53-15294800bd4b-credential-socket\") pod \"e2e-shared-a-455d3b-openshift-default-7b9767fd8c-wtg94\" (UID: \"3b6b3625-bfa7-4be3-9b53-15294800bd4b\") " pod="openshift-ingress/e2e-shared-a-455d3b-openshift-default-7b9767fd8c-wtg94" Jul 13 22:46:49.716063 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:46:49.715916 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3b6b3625-bfa7-4be3-9b53-15294800bd4b-istio-data\") pod \"e2e-shared-a-455d3b-openshift-default-7b9767fd8c-wtg94\" (UID: \"3b6b3625-bfa7-4be3-9b53-15294800bd4b\") " pod="openshift-ingress/e2e-shared-a-455d3b-openshift-default-7b9767fd8c-wtg94" Jul 13 22:46:49.716172 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:46:49.716109 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3b6b3625-bfa7-4be3-9b53-15294800bd4b-workload-certs\") pod \"e2e-shared-a-455d3b-openshift-default-7b9767fd8c-wtg94\" (UID: \"3b6b3625-bfa7-4be3-9b53-15294800bd4b\") " pod="openshift-ingress/e2e-shared-a-455d3b-openshift-default-7b9767fd8c-wtg94" Jul 13 22:46:49.716233 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:46:49.716181 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3b6b3625-bfa7-4be3-9b53-15294800bd4b-credential-socket\") pod \"e2e-shared-a-455d3b-openshift-default-7b9767fd8c-wtg94\" (UID: \"3b6b3625-bfa7-4be3-9b53-15294800bd4b\") " pod="openshift-ingress/e2e-shared-a-455d3b-openshift-default-7b9767fd8c-wtg94" Jul 13 22:46:49.716233 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:46:49.716219 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3b6b3625-bfa7-4be3-9b53-15294800bd4b-workload-socket\") pod \"e2e-shared-a-455d3b-openshift-default-7b9767fd8c-wtg94\" (UID: \"3b6b3625-bfa7-4be3-9b53-15294800bd4b\") " pod="openshift-ingress/e2e-shared-a-455d3b-openshift-default-7b9767fd8c-wtg94" Jul 13 22:46:49.716554 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:46:49.716524 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3b6b3625-bfa7-4be3-9b53-15294800bd4b-istiod-ca-cert\") pod \"e2e-shared-a-455d3b-openshift-default-7b9767fd8c-wtg94\" (UID: \"3b6b3625-bfa7-4be3-9b53-15294800bd4b\") " pod="openshift-ingress/e2e-shared-a-455d3b-openshift-default-7b9767fd8c-wtg94" Jul 13 22:46:49.718034 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:46:49.718012 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3b6b3625-bfa7-4be3-9b53-15294800bd4b-istio-envoy\") pod \"e2e-shared-a-455d3b-openshift-default-7b9767fd8c-wtg94\" (UID: \"3b6b3625-bfa7-4be3-9b53-15294800bd4b\") " pod="openshift-ingress/e2e-shared-a-455d3b-openshift-default-7b9767fd8c-wtg94" Jul 13 22:46:49.718339 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:46:49.718321 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3b6b3625-bfa7-4be3-9b53-15294800bd4b-istio-podinfo\") pod \"e2e-shared-a-455d3b-openshift-default-7b9767fd8c-wtg94\" (UID: \"3b6b3625-bfa7-4be3-9b53-15294800bd4b\") " pod="openshift-ingress/e2e-shared-a-455d3b-openshift-default-7b9767fd8c-wtg94" Jul 13 22:46:49.727114 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:46:49.727090 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5t626\" (UniqueName: \"kubernetes.io/projected/3b6b3625-bfa7-4be3-9b53-15294800bd4b-kube-api-access-5t626\") pod \"e2e-shared-a-455d3b-openshift-default-7b9767fd8c-wtg94\" (UID: \"3b6b3625-bfa7-4be3-9b53-15294800bd4b\") " pod="openshift-ingress/e2e-shared-a-455d3b-openshift-default-7b9767fd8c-wtg94" Jul 13 22:46:49.735616 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:46:49.735589 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3b6b3625-bfa7-4be3-9b53-15294800bd4b-istio-token\") pod \"e2e-shared-a-455d3b-openshift-default-7b9767fd8c-wtg94\" (UID: \"3b6b3625-bfa7-4be3-9b53-15294800bd4b\") " pod="openshift-ingress/e2e-shared-a-455d3b-openshift-default-7b9767fd8c-wtg94" Jul 13 22:46:49.808428 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:46:49.808391 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-455d3b-openshift-default-7b9767fd8c-wtg94" Jul 13 22:46:49.945097 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:46:49.945071 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-455d3b-openshift-default-7b9767fd8c-wtg94"] Jul 13 22:46:49.947141 ip-10-0-133-179 kubenswrapper[2583]: W0713 22:46:49.947119 2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3b6b3625_bfa7_4be3_9b53_15294800bd4b.slice/crio-c99a4b2e37a4e7cb3592794e10848cc21d16ffcce108750fdc7f2544ee3998af WatchSource:0}: Error finding container c99a4b2e37a4e7cb3592794e10848cc21d16ffcce108750fdc7f2544ee3998af: Status 404 returned error can't find the container with id c99a4b2e37a4e7cb3592794e10848cc21d16ffcce108750fdc7f2544ee3998af Jul 13 22:46:49.949256 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:46:49.949223 2583 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 13 22:46:49.949330 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:46:49.949289 2583 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 13 22:46:49.949330 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:46:49.949318 2583 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 13 22:46:50.546918 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:46:50.546883 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-455d3b-openshift-default-7b9767fd8c-wtg94" event={"ID":"3b6b3625-bfa7-4be3-9b53-15294800bd4b","Type":"ContainerStarted","Data":"cf932f47c2539acda802b6a97fb0738ff7d49664aad3e5655551b515e410f046"} Jul 13 22:46:50.547139 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:46:50.546942 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-455d3b-openshift-default-7b9767fd8c-wtg94" event={"ID":"3b6b3625-bfa7-4be3-9b53-15294800bd4b","Type":"ContainerStarted","Data":"c99a4b2e37a4e7cb3592794e10848cc21d16ffcce108750fdc7f2544ee3998af"} Jul 13 22:46:50.566096 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:46:50.566041 2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-455d3b-openshift-default-7b9767fd8c-wtg94" podStartSLOduration=1.566023054 podStartE2EDuration="1.566023054s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-13 22:46:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 22:46:50.565177089 +0000 UTC m=+2005.813092452" watchObservedRunningTime="2026-07-13 22:46:50.566023054 +0000 UTC m=+2005.813938404" Jul 13 22:46:50.809211 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:46:50.809095 2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-455d3b-openshift-default-7b9767fd8c-wtg94" Jul 13 22:46:51.813563 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:46:51.813535 2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-455d3b-openshift-default-7b9767fd8c-wtg94" Jul 13 22:46:52.556230 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:46:52.556200 2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-455d3b-openshift-default-7b9767fd8c-wtg94" Jul 13 22:46:52.557400 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:46:52.557382 2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-455d3b-openshift-default-7b9767fd8c-wtg94" Jul 13 22:46:52.564412 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:46:52.564384 2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-455d3b-7bb79b47cf-29n6r"] Jul 13 22:46:52.564690 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:46:52.564677 2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="950c3e6e-75a8-4e20-ba7f-84c0a08e1c1b" containerName="istio-proxy" Jul 13 22:46:52.564757 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:46:52.564691 2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="950c3e6e-75a8-4e20-ba7f-84c0a08e1c1b" containerName="istio-proxy" Jul 13 22:46:52.566687 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:46:52.566670 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-455d3b-7bb79b47cf-29n6r" Jul 13 22:46:52.569215 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:46:52.569198 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-a-455d3b-serving-cert\"" Jul 13 22:46:52.579427 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:46:52.579408 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-455d3b-7bb79b47cf-29n6r"] Jul 13 22:46:52.640098 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:46:52.640061 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tng45\" (UniqueName: \"kubernetes.io/projected/6558610a-5e68-4d35-977a-7992732426d8-kube-api-access-tng45\") pod \"maas-api-e2e-shared-a-455d3b-7bb79b47cf-29n6r\" (UID: \"6558610a-5e68-4d35-977a-7992732426d8\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-455d3b-7bb79b47cf-29n6r" Jul 13 22:46:52.640275 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:46:52.640256 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6558610a-5e68-4d35-977a-7992732426d8-openshift-service-ca\") pod \"maas-api-e2e-shared-a-455d3b-7bb79b47cf-29n6r\" (UID: \"6558610a-5e68-4d35-977a-7992732426d8\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-455d3b-7bb79b47cf-29n6r" Jul 13 22:46:52.640363 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:46:52.640333 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6558610a-5e68-4d35-977a-7992732426d8-maas-api-tls\") pod \"maas-api-e2e-shared-a-455d3b-7bb79b47cf-29n6r\" (UID: \"6558610a-5e68-4d35-977a-7992732426d8\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-455d3b-7bb79b47cf-29n6r" Jul 13 22:46:52.741286 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:46:52.741253 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6558610a-5e68-4d35-977a-7992732426d8-openshift-service-ca\") pod \"maas-api-e2e-shared-a-455d3b-7bb79b47cf-29n6r\" (UID: \"6558610a-5e68-4d35-977a-7992732426d8\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-455d3b-7bb79b47cf-29n6r" Jul 13 22:46:52.741462 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:46:52.741304 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6558610a-5e68-4d35-977a-7992732426d8-maas-api-tls\") pod \"maas-api-e2e-shared-a-455d3b-7bb79b47cf-29n6r\" (UID: \"6558610a-5e68-4d35-977a-7992732426d8\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-455d3b-7bb79b47cf-29n6r" Jul 13 22:46:52.741462 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:46:52.741371 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tng45\" (UniqueName: \"kubernetes.io/projected/6558610a-5e68-4d35-977a-7992732426d8-kube-api-access-tng45\") pod \"maas-api-e2e-shared-a-455d3b-7bb79b47cf-29n6r\" (UID: \"6558610a-5e68-4d35-977a-7992732426d8\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-455d3b-7bb79b47cf-29n6r" Jul 13 22:46:52.741539 ip-10-0-133-179 kubenswrapper[2583]: E0713 22:46:52.741485 2583 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-shared-a-455d3b-serving-cert: secret "maas-api-e2e-shared-a-455d3b-serving-cert" not found Jul 13 22:46:52.741599 ip-10-0-133-179 kubenswrapper[2583]: E0713 22:46:52.741552 2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6558610a-5e68-4d35-977a-7992732426d8-maas-api-tls podName:6558610a-5e68-4d35-977a-7992732426d8 nodeName:}" failed. No retries permitted until 2026-07-13 22:46:53.241531604 +0000 UTC m=+2008.489446945 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/6558610a-5e68-4d35-977a-7992732426d8-maas-api-tls") pod "maas-api-e2e-shared-a-455d3b-7bb79b47cf-29n6r" (UID: "6558610a-5e68-4d35-977a-7992732426d8") : secret "maas-api-e2e-shared-a-455d3b-serving-cert" not found Jul 13 22:46:52.742025 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:46:52.742008 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6558610a-5e68-4d35-977a-7992732426d8-openshift-service-ca\") pod \"maas-api-e2e-shared-a-455d3b-7bb79b47cf-29n6r\" (UID: \"6558610a-5e68-4d35-977a-7992732426d8\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-455d3b-7bb79b47cf-29n6r" Jul 13 22:46:52.752011 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:46:52.751984 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tng45\" (UniqueName: \"kubernetes.io/projected/6558610a-5e68-4d35-977a-7992732426d8-kube-api-access-tng45\") pod \"maas-api-e2e-shared-a-455d3b-7bb79b47cf-29n6r\" (UID: \"6558610a-5e68-4d35-977a-7992732426d8\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-455d3b-7bb79b47cf-29n6r" Jul 13 22:46:53.246746 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:46:53.246709 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6558610a-5e68-4d35-977a-7992732426d8-maas-api-tls\") pod \"maas-api-e2e-shared-a-455d3b-7bb79b47cf-29n6r\" (UID: \"6558610a-5e68-4d35-977a-7992732426d8\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-455d3b-7bb79b47cf-29n6r" Jul 13 22:46:53.249103 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:46:53.249083 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6558610a-5e68-4d35-977a-7992732426d8-maas-api-tls\") pod \"maas-api-e2e-shared-a-455d3b-7bb79b47cf-29n6r\" (UID: \"6558610a-5e68-4d35-977a-7992732426d8\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-455d3b-7bb79b47cf-29n6r" Jul 13 22:46:53.476962 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:46:53.476920 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-455d3b-7bb79b47cf-29n6r" Jul 13 22:46:53.613359 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:46:53.613333 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-455d3b-7bb79b47cf-29n6r"] Jul 13 22:46:53.616520 ip-10-0-133-179 kubenswrapper[2583]: W0713 22:46:53.616491 2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6558610a_5e68_4d35_977a_7992732426d8.slice/crio-667581107faae4140de9275f4c9d81c58dd4aed05d93e0d4a4c40dc1f2831155 WatchSource:0}: Error finding container 667581107faae4140de9275f4c9d81c58dd4aed05d93e0d4a4c40dc1f2831155: Status 404 returned error can't find the container with id 667581107faae4140de9275f4c9d81c58dd4aed05d93e0d4a4c40dc1f2831155 Jul 13 22:46:53.857357 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:46:53.857333 2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-1276a7-openshift-default-84f5889fb9-2qzdr" Jul 13 22:46:53.954400 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:46:53.954369 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qdpwj\" (UniqueName: \"kubernetes.io/projected/0b5a60d1-133b-4e0b-b868-fbd4adc706db-kube-api-access-qdpwj\") pod \"0b5a60d1-133b-4e0b-b868-fbd4adc706db\" (UID: \"0b5a60d1-133b-4e0b-b868-fbd4adc706db\") " Jul 13 22:46:53.954589 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:46:53.954425 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0b5a60d1-133b-4e0b-b868-fbd4adc706db-istio-podinfo\") pod \"0b5a60d1-133b-4e0b-b868-fbd4adc706db\" (UID: \"0b5a60d1-133b-4e0b-b868-fbd4adc706db\") " Jul 13 22:46:53.954589 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:46:53.954450 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0b5a60d1-133b-4e0b-b868-fbd4adc706db-istio-token\") pod \"0b5a60d1-133b-4e0b-b868-fbd4adc706db\" (UID: \"0b5a60d1-133b-4e0b-b868-fbd4adc706db\") " Jul 13 22:46:53.954589 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:46:53.954472 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0b5a60d1-133b-4e0b-b868-fbd4adc706db-istio-envoy\") pod \"0b5a60d1-133b-4e0b-b868-fbd4adc706db\" (UID: \"0b5a60d1-133b-4e0b-b868-fbd4adc706db\") " Jul 13 22:46:53.954589 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:46:53.954500 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0b5a60d1-133b-4e0b-b868-fbd4adc706db-istio-data\") pod \"0b5a60d1-133b-4e0b-b868-fbd4adc706db\" (UID: \"0b5a60d1-133b-4e0b-b868-fbd4adc706db\") " Jul 13 22:46:53.954589 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:46:53.954526 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0b5a60d1-133b-4e0b-b868-fbd4adc706db-workload-certs\") pod \"0b5a60d1-133b-4e0b-b868-fbd4adc706db\" (UID: \"0b5a60d1-133b-4e0b-b868-fbd4adc706db\") " Jul 13 22:46:53.954589 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:46:53.954544 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0b5a60d1-133b-4e0b-b868-fbd4adc706db-workload-socket\") pod \"0b5a60d1-133b-4e0b-b868-fbd4adc706db\" (UID: \"0b5a60d1-133b-4e0b-b868-fbd4adc706db\") " Jul 13 22:46:53.954912 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:46:53.954597 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0b5a60d1-133b-4e0b-b868-fbd4adc706db-istiod-ca-cert\") pod \"0b5a60d1-133b-4e0b-b868-fbd4adc706db\" (UID: \"0b5a60d1-133b-4e0b-b868-fbd4adc706db\") " Jul 13 22:46:53.954912 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:46:53.954653 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0b5a60d1-133b-4e0b-b868-fbd4adc706db-credential-socket\") pod \"0b5a60d1-133b-4e0b-b868-fbd4adc706db\" (UID: \"0b5a60d1-133b-4e0b-b868-fbd4adc706db\") " Jul 13 22:46:53.955028 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:46:53.954981 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0b5a60d1-133b-4e0b-b868-fbd4adc706db-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "0b5a60d1-133b-4e0b-b868-fbd4adc706db" (UID: "0b5a60d1-133b-4e0b-b868-fbd4adc706db"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 13 22:46:53.955084 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:46:53.955066 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0b5a60d1-133b-4e0b-b868-fbd4adc706db-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "0b5a60d1-133b-4e0b-b868-fbd4adc706db" (UID: "0b5a60d1-133b-4e0b-b868-fbd4adc706db"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 13 22:46:53.955140 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:46:53.955075 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0b5a60d1-133b-4e0b-b868-fbd4adc706db-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "0b5a60d1-133b-4e0b-b868-fbd4adc706db" (UID: "0b5a60d1-133b-4e0b-b868-fbd4adc706db"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 13 22:46:53.955386 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:46:53.955361 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0b5a60d1-133b-4e0b-b868-fbd4adc706db-istio-data" (OuterVolumeSpecName: "istio-data") pod "0b5a60d1-133b-4e0b-b868-fbd4adc706db" (UID: "0b5a60d1-133b-4e0b-b868-fbd4adc706db"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 13 22:46:53.955483 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:46:53.955417 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0b5a60d1-133b-4e0b-b868-fbd4adc706db-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "0b5a60d1-133b-4e0b-b868-fbd4adc706db" (UID: "0b5a60d1-133b-4e0b-b868-fbd4adc706db"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 13 22:46:53.956677 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:46:53.956646 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0b5a60d1-133b-4e0b-b868-fbd4adc706db-kube-api-access-qdpwj" (OuterVolumeSpecName: "kube-api-access-qdpwj") pod "0b5a60d1-133b-4e0b-b868-fbd4adc706db" (UID: "0b5a60d1-133b-4e0b-b868-fbd4adc706db"). InnerVolumeSpecName "kube-api-access-qdpwj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 13 22:46:53.956823 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:46:53.956800 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/0b5a60d1-133b-4e0b-b868-fbd4adc706db-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "0b5a60d1-133b-4e0b-b868-fbd4adc706db" (UID: "0b5a60d1-133b-4e0b-b868-fbd4adc706db"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 13 22:46:53.957119 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:46:53.957095 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0b5a60d1-133b-4e0b-b868-fbd4adc706db-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "0b5a60d1-133b-4e0b-b868-fbd4adc706db" (UID: "0b5a60d1-133b-4e0b-b868-fbd4adc706db"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 13 22:46:53.957119 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:46:53.957103 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0b5a60d1-133b-4e0b-b868-fbd4adc706db-istio-token" (OuterVolumeSpecName: "istio-token") pod "0b5a60d1-133b-4e0b-b868-fbd4adc706db" (UID: "0b5a60d1-133b-4e0b-b868-fbd4adc706db"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 13 22:46:54.055517 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:46:54.055487 2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qdpwj\" (UniqueName: \"kubernetes.io/projected/0b5a60d1-133b-4e0b-b868-fbd4adc706db-kube-api-access-qdpwj\") on node \"ip-10-0-133-179.ec2.internal\" DevicePath \"\"" Jul 13 22:46:54.055517 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:46:54.055514 2583 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0b5a60d1-133b-4e0b-b868-fbd4adc706db-istio-podinfo\") on node \"ip-10-0-133-179.ec2.internal\" DevicePath \"\"" Jul 13 22:46:54.055517 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:46:54.055524 2583 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0b5a60d1-133b-4e0b-b868-fbd4adc706db-istio-token\") on node \"ip-10-0-133-179.ec2.internal\" DevicePath \"\"" Jul 13 22:46:54.055779 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:46:54.055534 2583 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0b5a60d1-133b-4e0b-b868-fbd4adc706db-istio-envoy\") on node \"ip-10-0-133-179.ec2.internal\" DevicePath \"\"" Jul 13 22:46:54.055779 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:46:54.055541 2583 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0b5a60d1-133b-4e0b-b868-fbd4adc706db-istio-data\") on node \"ip-10-0-133-179.ec2.internal\" DevicePath \"\"" Jul 13 22:46:54.055779 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:46:54.055549 2583 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0b5a60d1-133b-4e0b-b868-fbd4adc706db-workload-certs\") on node \"ip-10-0-133-179.ec2.internal\" DevicePath \"\"" Jul 13 22:46:54.055779 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:46:54.055557 2583 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0b5a60d1-133b-4e0b-b868-fbd4adc706db-workload-socket\") on node \"ip-10-0-133-179.ec2.internal\" DevicePath \"\"" Jul 13 22:46:54.055779 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:46:54.055582 2583 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0b5a60d1-133b-4e0b-b868-fbd4adc706db-istiod-ca-cert\") on node \"ip-10-0-133-179.ec2.internal\" DevicePath \"\"" Jul 13 22:46:54.055779 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:46:54.055591 2583 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0b5a60d1-133b-4e0b-b868-fbd4adc706db-credential-socket\") on node \"ip-10-0-133-179.ec2.internal\" DevicePath \"\"" Jul 13 22:46:54.565057 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:46:54.565021 2583 generic.go:356] "Generic (PLEG): container finished" podID="0b5a60d1-133b-4e0b-b868-fbd4adc706db" containerID="14bb5b2bf3bbf2709f7a53e1378c91164d73b284a5a825aad2f078daa401e66d" exitCode=0 Jul 13 22:46:54.565529 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:46:54.565081 2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-1276a7-openshift-default-84f5889fb9-2qzdr" Jul 13 22:46:54.565529 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:46:54.565109 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-1276a7-openshift-default-84f5889fb9-2qzdr" event={"ID":"0b5a60d1-133b-4e0b-b868-fbd4adc706db","Type":"ContainerDied","Data":"14bb5b2bf3bbf2709f7a53e1378c91164d73b284a5a825aad2f078daa401e66d"} Jul 13 22:46:54.565529 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:46:54.565166 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-1276a7-openshift-default-84f5889fb9-2qzdr" event={"ID":"0b5a60d1-133b-4e0b-b868-fbd4adc706db","Type":"ContainerDied","Data":"3fb5a8a7164f11e3ce1b914e8a7f3cb30b5ecbb5cca71083f6761d8966662371"} Jul 13 22:46:54.565529 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:46:54.565194 2583 scope.go:117] "RemoveContainer" containerID="14bb5b2bf3bbf2709f7a53e1378c91164d73b284a5a825aad2f078daa401e66d" Jul 13 22:46:54.566684 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:46:54.566653 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-455d3b-7bb79b47cf-29n6r" event={"ID":"6558610a-5e68-4d35-977a-7992732426d8","Type":"ContainerStarted","Data":"09987f1c15253b6d3143083c3a45c1bf22bd0c34dada317acafe8a0a4fee0ccd"} Jul 13 22:46:54.566768 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:46:54.566707 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-455d3b-7bb79b47cf-29n6r" event={"ID":"6558610a-5e68-4d35-977a-7992732426d8","Type":"ContainerStarted","Data":"667581107faae4140de9275f4c9d81c58dd4aed05d93e0d4a4c40dc1f2831155"} Jul 13 22:46:54.566768 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:46:54.566762 2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-455d3b-7bb79b47cf-29n6r" Jul 13 22:46:54.573820 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:46:54.573806 2583 scope.go:117] "RemoveContainer" containerID="b98188f01ac9ee00971d7d1060c4934b04e42a7424cc8cf322450e6cf55c4dff" Jul 13 22:46:54.581272 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:46:54.581255 2583 scope.go:117] "RemoveContainer" containerID="14bb5b2bf3bbf2709f7a53e1378c91164d73b284a5a825aad2f078daa401e66d" Jul 13 22:46:54.581530 ip-10-0-133-179 kubenswrapper[2583]: E0713 22:46:54.581513 2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"14bb5b2bf3bbf2709f7a53e1378c91164d73b284a5a825aad2f078daa401e66d\": container with ID starting with 14bb5b2bf3bbf2709f7a53e1378c91164d73b284a5a825aad2f078daa401e66d not found: ID does not exist" containerID="14bb5b2bf3bbf2709f7a53e1378c91164d73b284a5a825aad2f078daa401e66d" Jul 13 22:46:54.581602 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:46:54.581540 2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"14bb5b2bf3bbf2709f7a53e1378c91164d73b284a5a825aad2f078daa401e66d"} err="failed to get container status \"14bb5b2bf3bbf2709f7a53e1378c91164d73b284a5a825aad2f078daa401e66d\": rpc error: code = NotFound desc = could not find container \"14bb5b2bf3bbf2709f7a53e1378c91164d73b284a5a825aad2f078daa401e66d\": container with ID starting with 14bb5b2bf3bbf2709f7a53e1378c91164d73b284a5a825aad2f078daa401e66d not found: ID does not exist" Jul 13 22:46:54.581602 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:46:54.581557 2583 scope.go:117] "RemoveContainer" containerID="b98188f01ac9ee00971d7d1060c4934b04e42a7424cc8cf322450e6cf55c4dff" Jul 13 22:46:54.581837 ip-10-0-133-179 kubenswrapper[2583]: E0713 22:46:54.581816 2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b98188f01ac9ee00971d7d1060c4934b04e42a7424cc8cf322450e6cf55c4dff\": container with ID starting with b98188f01ac9ee00971d7d1060c4934b04e42a7424cc8cf322450e6cf55c4dff not found: ID does not exist" containerID="b98188f01ac9ee00971d7d1060c4934b04e42a7424cc8cf322450e6cf55c4dff" Jul 13 22:46:54.581875 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:46:54.581844 2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b98188f01ac9ee00971d7d1060c4934b04e42a7424cc8cf322450e6cf55c4dff"} err="failed to get container status \"b98188f01ac9ee00971d7d1060c4934b04e42a7424cc8cf322450e6cf55c4dff\": rpc error: code = NotFound desc = could not find container \"b98188f01ac9ee00971d7d1060c4934b04e42a7424cc8cf322450e6cf55c4dff\": container with ID starting with b98188f01ac9ee00971d7d1060c4934b04e42a7424cc8cf322450e6cf55c4dff not found: ID does not exist" Jul 13 22:46:54.585811 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:46:54.585763 2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-455d3b-7bb79b47cf-29n6r" podStartSLOduration=2.229509208 podStartE2EDuration="2.585747922s" totalImagesPullingTime="356.238714ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-13 22:46:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 22:46:54.585243981 +0000 UTC m=+2009.833159331" watchObservedRunningTime="2026-07-13 22:46:54.585747922 +0000 UTC m=+2009.833663272" Jul 13 22:46:54.598643 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:46:54.598618 2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-1276a7-openshift-default-84f5889fb9-2qzdr"] Jul 13 22:46:54.602260 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:46:54.602238 2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-b-1276a7-openshift-default-84f5889fb9-2qzdr"] Jul 13 22:46:55.389689 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:46:55.389655 2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0b5a60d1-133b-4e0b-b868-fbd4adc706db" path="/var/lib/kubelet/pods/0b5a60d1-133b-4e0b-b868-fbd4adc706db/volumes" Jul 13 22:46:57.628695 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:46:57.628659 2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-b37782-openshift-default-59c894cff-6xtjx"] Jul 13 22:46:57.629074 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:46:57.628978 2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0b5a60d1-133b-4e0b-b868-fbd4adc706db" containerName="istio-proxy" Jul 13 22:46:57.629074 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:46:57.628989 2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="0b5a60d1-133b-4e0b-b868-fbd4adc706db" containerName="istio-proxy" Jul 13 22:46:57.629074 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:46:57.629054 2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="0b5a60d1-133b-4e0b-b868-fbd4adc706db" containerName="istio-proxy" Jul 13 22:46:57.631909 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:46:57.631891 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-b37782-openshift-default-59c894cff-6xtjx" Jul 13 22:46:57.634547 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:46:57.634527 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-b37782-openshift-default-dockercfg-2cv5h\"" Jul 13 22:46:57.660020 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:46:57.659989 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-b37782-openshift-default-59c894cff-6xtjx"] Jul 13 22:46:57.687386 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:46:57.687355 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8d9ee87d-31e3-459a-9510-9e985b2359bb-istio-token\") pod \"e2e-shared-b-b37782-openshift-default-59c894cff-6xtjx\" (UID: \"8d9ee87d-31e3-459a-9510-9e985b2359bb\") " pod="openshift-ingress/e2e-shared-b-b37782-openshift-default-59c894cff-6xtjx" Jul 13 22:46:57.687602 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:46:57.687405 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8d9ee87d-31e3-459a-9510-9e985b2359bb-workload-socket\") pod \"e2e-shared-b-b37782-openshift-default-59c894cff-6xtjx\" (UID: \"8d9ee87d-31e3-459a-9510-9e985b2359bb\") " pod="openshift-ingress/e2e-shared-b-b37782-openshift-default-59c894cff-6xtjx" Jul 13 22:46:57.687602 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:46:57.687495 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8d9ee87d-31e3-459a-9510-9e985b2359bb-istio-envoy\") pod \"e2e-shared-b-b37782-openshift-default-59c894cff-6xtjx\" (UID: \"8d9ee87d-31e3-459a-9510-9e985b2359bb\") " pod="openshift-ingress/e2e-shared-b-b37782-openshift-default-59c894cff-6xtjx" Jul 13 22:46:57.687602 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:46:57.687536 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8d9ee87d-31e3-459a-9510-9e985b2359bb-workload-certs\") pod \"e2e-shared-b-b37782-openshift-default-59c894cff-6xtjx\" (UID: \"8d9ee87d-31e3-459a-9510-9e985b2359bb\") " pod="openshift-ingress/e2e-shared-b-b37782-openshift-default-59c894cff-6xtjx" Jul 13 22:46:57.687798 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:46:57.687605 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8d9ee87d-31e3-459a-9510-9e985b2359bb-istio-data\") pod \"e2e-shared-b-b37782-openshift-default-59c894cff-6xtjx\" (UID: \"8d9ee87d-31e3-459a-9510-9e985b2359bb\") " pod="openshift-ingress/e2e-shared-b-b37782-openshift-default-59c894cff-6xtjx" Jul 13 22:46:57.687798 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:46:57.687643 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8d9ee87d-31e3-459a-9510-9e985b2359bb-credential-socket\") pod \"e2e-shared-b-b37782-openshift-default-59c894cff-6xtjx\" (UID: \"8d9ee87d-31e3-459a-9510-9e985b2359bb\") " pod="openshift-ingress/e2e-shared-b-b37782-openshift-default-59c894cff-6xtjx" Jul 13 22:46:57.687798 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:46:57.687665 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8d9ee87d-31e3-459a-9510-9e985b2359bb-istio-podinfo\") pod \"e2e-shared-b-b37782-openshift-default-59c894cff-6xtjx\" (UID: \"8d9ee87d-31e3-459a-9510-9e985b2359bb\") " pod="openshift-ingress/e2e-shared-b-b37782-openshift-default-59c894cff-6xtjx" Jul 13 22:46:57.687798 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:46:57.687685 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8d9ee87d-31e3-459a-9510-9e985b2359bb-istiod-ca-cert\") pod \"e2e-shared-b-b37782-openshift-default-59c894cff-6xtjx\" (UID: \"8d9ee87d-31e3-459a-9510-9e985b2359bb\") " pod="openshift-ingress/e2e-shared-b-b37782-openshift-default-59c894cff-6xtjx" Jul 13 22:46:57.687798 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:46:57.687719 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fvmvc\" (UniqueName: \"kubernetes.io/projected/8d9ee87d-31e3-459a-9510-9e985b2359bb-kube-api-access-fvmvc\") pod \"e2e-shared-b-b37782-openshift-default-59c894cff-6xtjx\" (UID: \"8d9ee87d-31e3-459a-9510-9e985b2359bb\") " pod="openshift-ingress/e2e-shared-b-b37782-openshift-default-59c894cff-6xtjx" Jul 13 22:46:57.788430 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:46:57.788372 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8d9ee87d-31e3-459a-9510-9e985b2359bb-workload-socket\") pod \"e2e-shared-b-b37782-openshift-default-59c894cff-6xtjx\" (UID: \"8d9ee87d-31e3-459a-9510-9e985b2359bb\") " pod="openshift-ingress/e2e-shared-b-b37782-openshift-default-59c894cff-6xtjx" Jul 13 22:46:57.788430 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:46:57.788437 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8d9ee87d-31e3-459a-9510-9e985b2359bb-istio-envoy\") pod \"e2e-shared-b-b37782-openshift-default-59c894cff-6xtjx\" (UID: \"8d9ee87d-31e3-459a-9510-9e985b2359bb\") " pod="openshift-ingress/e2e-shared-b-b37782-openshift-default-59c894cff-6xtjx" Jul 13 22:46:57.788737 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:46:57.788459 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8d9ee87d-31e3-459a-9510-9e985b2359bb-workload-certs\") pod \"e2e-shared-b-b37782-openshift-default-59c894cff-6xtjx\" (UID: \"8d9ee87d-31e3-459a-9510-9e985b2359bb\") " pod="openshift-ingress/e2e-shared-b-b37782-openshift-default-59c894cff-6xtjx" Jul 13 22:46:57.788737 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:46:57.788476 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8d9ee87d-31e3-459a-9510-9e985b2359bb-istio-data\") pod \"e2e-shared-b-b37782-openshift-default-59c894cff-6xtjx\" (UID: \"8d9ee87d-31e3-459a-9510-9e985b2359bb\") " pod="openshift-ingress/e2e-shared-b-b37782-openshift-default-59c894cff-6xtjx" Jul 13 22:46:57.788737 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:46:57.788501 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8d9ee87d-31e3-459a-9510-9e985b2359bb-credential-socket\") pod \"e2e-shared-b-b37782-openshift-default-59c894cff-6xtjx\" (UID: \"8d9ee87d-31e3-459a-9510-9e985b2359bb\") " pod="openshift-ingress/e2e-shared-b-b37782-openshift-default-59c894cff-6xtjx" Jul 13 22:46:57.788737 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:46:57.788516 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8d9ee87d-31e3-459a-9510-9e985b2359bb-istio-podinfo\") pod \"e2e-shared-b-b37782-openshift-default-59c894cff-6xtjx\" (UID: \"8d9ee87d-31e3-459a-9510-9e985b2359bb\") " pod="openshift-ingress/e2e-shared-b-b37782-openshift-default-59c894cff-6xtjx" Jul 13 22:46:57.788737 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:46:57.788536 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8d9ee87d-31e3-459a-9510-9e985b2359bb-istiod-ca-cert\") pod \"e2e-shared-b-b37782-openshift-default-59c894cff-6xtjx\" (UID: \"8d9ee87d-31e3-459a-9510-9e985b2359bb\") " pod="openshift-ingress/e2e-shared-b-b37782-openshift-default-59c894cff-6xtjx" Jul 13 22:46:57.788737 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:46:57.788556 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fvmvc\" (UniqueName: \"kubernetes.io/projected/8d9ee87d-31e3-459a-9510-9e985b2359bb-kube-api-access-fvmvc\") pod \"e2e-shared-b-b37782-openshift-default-59c894cff-6xtjx\" (UID: \"8d9ee87d-31e3-459a-9510-9e985b2359bb\") " pod="openshift-ingress/e2e-shared-b-b37782-openshift-default-59c894cff-6xtjx" Jul 13 22:46:57.788737 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:46:57.788660 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8d9ee87d-31e3-459a-9510-9e985b2359bb-istio-token\") pod \"e2e-shared-b-b37782-openshift-default-59c894cff-6xtjx\" (UID: \"8d9ee87d-31e3-459a-9510-9e985b2359bb\") " pod="openshift-ingress/e2e-shared-b-b37782-openshift-default-59c894cff-6xtjx" Jul 13 22:46:57.789118 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:46:57.788999 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8d9ee87d-31e3-459a-9510-9e985b2359bb-credential-socket\") pod \"e2e-shared-b-b37782-openshift-default-59c894cff-6xtjx\" (UID: \"8d9ee87d-31e3-459a-9510-9e985b2359bb\") " pod="openshift-ingress/e2e-shared-b-b37782-openshift-default-59c894cff-6xtjx" Jul 13 22:46:57.789118 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:46:57.789029 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8d9ee87d-31e3-459a-9510-9e985b2359bb-workload-certs\") pod \"e2e-shared-b-b37782-openshift-default-59c894cff-6xtjx\" (UID: \"8d9ee87d-31e3-459a-9510-9e985b2359bb\") " pod="openshift-ingress/e2e-shared-b-b37782-openshift-default-59c894cff-6xtjx" Jul 13 22:46:57.789227 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:46:57.789192 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8d9ee87d-31e3-459a-9510-9e985b2359bb-istio-data\") pod \"e2e-shared-b-b37782-openshift-default-59c894cff-6xtjx\" (UID: \"8d9ee87d-31e3-459a-9510-9e985b2359bb\") " pod="openshift-ingress/e2e-shared-b-b37782-openshift-default-59c894cff-6xtjx" Jul 13 22:46:57.789290 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:46:57.789248 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8d9ee87d-31e3-459a-9510-9e985b2359bb-workload-socket\") pod \"e2e-shared-b-b37782-openshift-default-59c894cff-6xtjx\" (UID: \"8d9ee87d-31e3-459a-9510-9e985b2359bb\") " pod="openshift-ingress/e2e-shared-b-b37782-openshift-default-59c894cff-6xtjx" Jul 13 22:46:57.789713 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:46:57.789692 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8d9ee87d-31e3-459a-9510-9e985b2359bb-istiod-ca-cert\") pod \"e2e-shared-b-b37782-openshift-default-59c894cff-6xtjx\" (UID: \"8d9ee87d-31e3-459a-9510-9e985b2359bb\") " pod="openshift-ingress/e2e-shared-b-b37782-openshift-default-59c894cff-6xtjx" Jul 13 22:46:57.790927 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:46:57.790907 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8d9ee87d-31e3-459a-9510-9e985b2359bb-istio-envoy\") pod \"e2e-shared-b-b37782-openshift-default-59c894cff-6xtjx\" (UID: \"8d9ee87d-31e3-459a-9510-9e985b2359bb\") " pod="openshift-ingress/e2e-shared-b-b37782-openshift-default-59c894cff-6xtjx" Jul 13 22:46:57.791132 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:46:57.791112 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8d9ee87d-31e3-459a-9510-9e985b2359bb-istio-podinfo\") pod \"e2e-shared-b-b37782-openshift-default-59c894cff-6xtjx\" (UID: \"8d9ee87d-31e3-459a-9510-9e985b2359bb\") " pod="openshift-ingress/e2e-shared-b-b37782-openshift-default-59c894cff-6xtjx" Jul 13 22:46:57.797678 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:46:57.797644 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fvmvc\" (UniqueName: \"kubernetes.io/projected/8d9ee87d-31e3-459a-9510-9e985b2359bb-kube-api-access-fvmvc\") pod \"e2e-shared-b-b37782-openshift-default-59c894cff-6xtjx\" (UID: \"8d9ee87d-31e3-459a-9510-9e985b2359bb\") " pod="openshift-ingress/e2e-shared-b-b37782-openshift-default-59c894cff-6xtjx" Jul 13 22:46:57.798081 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:46:57.798057 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8d9ee87d-31e3-459a-9510-9e985b2359bb-istio-token\") pod \"e2e-shared-b-b37782-openshift-default-59c894cff-6xtjx\" (UID: \"8d9ee87d-31e3-459a-9510-9e985b2359bb\") " pod="openshift-ingress/e2e-shared-b-b37782-openshift-default-59c894cff-6xtjx" Jul 13 22:46:57.944439 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:46:57.944343 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-b37782-openshift-default-59c894cff-6xtjx" Jul 13 22:46:58.099199 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:46:58.099166 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-b37782-openshift-default-59c894cff-6xtjx"] Jul 13 22:46:58.100780 ip-10-0-133-179 kubenswrapper[2583]: W0713 22:46:58.100753 2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8d9ee87d_31e3_459a_9510_9e985b2359bb.slice/crio-b6710a251fe80bb1714e1d7c2db45451846151f5641dd353d96666866198b0c7 WatchSource:0}: Error finding container b6710a251fe80bb1714e1d7c2db45451846151f5641dd353d96666866198b0c7: Status 404 returned error can't find the container with id b6710a251fe80bb1714e1d7c2db45451846151f5641dd353d96666866198b0c7 Jul 13 22:46:58.103332 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:46:58.103282 2583 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 13 22:46:58.103480 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:46:58.103368 2583 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 13 22:46:58.103480 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:46:58.103412 2583 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 13 22:46:58.586375 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:46:58.586338 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-b37782-openshift-default-59c894cff-6xtjx" event={"ID":"8d9ee87d-31e3-459a-9510-9e985b2359bb","Type":"ContainerStarted","Data":"47c0d5c976c45a8bab8d14651a5400ae83ee4bb9d5e8d4a59362aa1fa80ad960"} Jul 13 22:46:58.586559 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:46:58.586397 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-b37782-openshift-default-59c894cff-6xtjx" event={"ID":"8d9ee87d-31e3-459a-9510-9e985b2359bb","Type":"ContainerStarted","Data":"b6710a251fe80bb1714e1d7c2db45451846151f5641dd353d96666866198b0c7"} Jul 13 22:46:58.608443 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:46:58.608390 2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-b37782-openshift-default-59c894cff-6xtjx" podStartSLOduration=1.6083748820000001 podStartE2EDuration="1.608374882s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-13 22:46:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 22:46:58.605881074 +0000 UTC m=+2013.853796437" watchObservedRunningTime="2026-07-13 22:46:58.608374882 +0000 UTC m=+2013.856290232" Jul 13 22:46:58.945003 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:46:58.944896 2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-b37782-openshift-default-59c894cff-6xtjx" Jul 13 22:46:59.953127 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:46:59.953087 2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-b37782-openshift-default-59c894cff-6xtjx" Jul 13 22:47:00.578523 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:47:00.578487 2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-455d3b-7bb79b47cf-29n6r" Jul 13 22:47:00.595198 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:47:00.595169 2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-b37782-openshift-default-59c894cff-6xtjx" Jul 13 22:47:00.596136 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:47:00.596120 2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-b37782-openshift-default-59c894cff-6xtjx" Jul 13 22:47:00.741592 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:47:00.740443 2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-b37782-84fd897778-24bxc"] Jul 13 22:47:00.741592 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:47:00.740926 2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0b5a60d1-133b-4e0b-b868-fbd4adc706db" containerName="istio-proxy" Jul 13 22:47:00.741592 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:47:00.740943 2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="0b5a60d1-133b-4e0b-b868-fbd4adc706db" containerName="istio-proxy" Jul 13 22:47:00.741592 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:47:00.741037 2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="0b5a60d1-133b-4e0b-b868-fbd4adc706db" containerName="istio-proxy" Jul 13 22:47:00.744486 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:47:00.743748 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-b37782-84fd897778-24bxc" Jul 13 22:47:00.753599 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:47:00.750456 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-b-b37782-serving-cert\"" Jul 13 22:47:00.765396 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:47:00.755489 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-b37782-84fd897778-24bxc"] Jul 13 22:47:00.815815 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:47:00.815778 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a01fedb6-f2c1-413b-96dd-50beeefeb079-maas-api-tls\") pod \"maas-api-e2e-shared-b-b37782-84fd897778-24bxc\" (UID: \"a01fedb6-f2c1-413b-96dd-50beeefeb079\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-b37782-84fd897778-24bxc" Jul 13 22:47:00.815815 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:47:00.815823 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jb59q\" (UniqueName: \"kubernetes.io/projected/a01fedb6-f2c1-413b-96dd-50beeefeb079-kube-api-access-jb59q\") pod \"maas-api-e2e-shared-b-b37782-84fd897778-24bxc\" (UID: \"a01fedb6-f2c1-413b-96dd-50beeefeb079\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-b37782-84fd897778-24bxc" Jul 13 22:47:00.816045 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:47:00.815915 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a01fedb6-f2c1-413b-96dd-50beeefeb079-openshift-service-ca\") pod \"maas-api-e2e-shared-b-b37782-84fd897778-24bxc\" (UID: \"a01fedb6-f2c1-413b-96dd-50beeefeb079\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-b37782-84fd897778-24bxc" Jul 13 22:47:00.916815 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:47:00.916713 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a01fedb6-f2c1-413b-96dd-50beeefeb079-maas-api-tls\") pod \"maas-api-e2e-shared-b-b37782-84fd897778-24bxc\" (UID: \"a01fedb6-f2c1-413b-96dd-50beeefeb079\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-b37782-84fd897778-24bxc" Jul 13 22:47:00.916815 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:47:00.916758 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jb59q\" (UniqueName: \"kubernetes.io/projected/a01fedb6-f2c1-413b-96dd-50beeefeb079-kube-api-access-jb59q\") pod \"maas-api-e2e-shared-b-b37782-84fd897778-24bxc\" (UID: \"a01fedb6-f2c1-413b-96dd-50beeefeb079\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-b37782-84fd897778-24bxc" Jul 13 22:47:00.916815 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:47:00.916794 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a01fedb6-f2c1-413b-96dd-50beeefeb079-openshift-service-ca\") pod \"maas-api-e2e-shared-b-b37782-84fd897778-24bxc\" (UID: \"a01fedb6-f2c1-413b-96dd-50beeefeb079\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-b37782-84fd897778-24bxc" Jul 13 22:47:00.917105 ip-10-0-133-179 kubenswrapper[2583]: E0713 22:47:00.916883 2583 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-shared-b-b37782-serving-cert: secret "maas-api-e2e-shared-b-b37782-serving-cert" not found Jul 13 22:47:00.917105 ip-10-0-133-179 kubenswrapper[2583]: E0713 22:47:00.916959 2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a01fedb6-f2c1-413b-96dd-50beeefeb079-maas-api-tls podName:a01fedb6-f2c1-413b-96dd-50beeefeb079 nodeName:}" failed. No retries permitted until 2026-07-13 22:47:01.416937523 +0000 UTC m=+2016.664852851 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/a01fedb6-f2c1-413b-96dd-50beeefeb079-maas-api-tls") pod "maas-api-e2e-shared-b-b37782-84fd897778-24bxc" (UID: "a01fedb6-f2c1-413b-96dd-50beeefeb079") : secret "maas-api-e2e-shared-b-b37782-serving-cert" not found Jul 13 22:47:00.917411 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:47:00.917392 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a01fedb6-f2c1-413b-96dd-50beeefeb079-openshift-service-ca\") pod \"maas-api-e2e-shared-b-b37782-84fd897778-24bxc\" (UID: \"a01fedb6-f2c1-413b-96dd-50beeefeb079\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-b37782-84fd897778-24bxc" Jul 13 22:47:00.936030 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:47:00.935988 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jb59q\" (UniqueName: \"kubernetes.io/projected/a01fedb6-f2c1-413b-96dd-50beeefeb079-kube-api-access-jb59q\") pod \"maas-api-e2e-shared-b-b37782-84fd897778-24bxc\" (UID: \"a01fedb6-f2c1-413b-96dd-50beeefeb079\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-b37782-84fd897778-24bxc" Jul 13 22:47:01.422324 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:47:01.422292 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a01fedb6-f2c1-413b-96dd-50beeefeb079-maas-api-tls\") pod \"maas-api-e2e-shared-b-b37782-84fd897778-24bxc\" (UID: \"a01fedb6-f2c1-413b-96dd-50beeefeb079\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-b37782-84fd897778-24bxc" Jul 13 22:47:01.424795 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:47:01.424772 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a01fedb6-f2c1-413b-96dd-50beeefeb079-maas-api-tls\") pod \"maas-api-e2e-shared-b-b37782-84fd897778-24bxc\" (UID: \"a01fedb6-f2c1-413b-96dd-50beeefeb079\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-b37782-84fd897778-24bxc" Jul 13 22:47:01.671891 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:47:01.671847 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-b37782-84fd897778-24bxc" Jul 13 22:47:01.826013 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:47:01.825984 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-b37782-84fd897778-24bxc"] Jul 13 22:47:01.828647 ip-10-0-133-179 kubenswrapper[2583]: W0713 22:47:01.828613 2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda01fedb6_f2c1_413b_96dd_50beeefeb079.slice/crio-35652982f9eb62ff427eec028a2849807e6f37600013de26a2e78b4c64d5f485 WatchSource:0}: Error finding container 35652982f9eb62ff427eec028a2849807e6f37600013de26a2e78b4c64d5f485: Status 404 returned error can't find the container with id 35652982f9eb62ff427eec028a2849807e6f37600013de26a2e78b4c64d5f485 Jul 13 22:47:02.605161 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:47:02.605125 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-b37782-84fd897778-24bxc" event={"ID":"a01fedb6-f2c1-413b-96dd-50beeefeb079","Type":"ContainerStarted","Data":"4e59daf043e268abb4eec99c90eed2b07326f085e999b64101413a64b852b3d4"} Jul 13 22:47:02.605609 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:47:02.605187 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-b37782-84fd897778-24bxc" event={"ID":"a01fedb6-f2c1-413b-96dd-50beeefeb079","Type":"ContainerStarted","Data":"35652982f9eb62ff427eec028a2849807e6f37600013de26a2e78b4c64d5f485"} Jul 13 22:47:02.605609 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:47:02.605350 2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-b37782-84fd897778-24bxc" Jul 13 22:47:02.622812 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:47:02.622754 2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-b37782-84fd897778-24bxc" podStartSLOduration=2.272234706 podStartE2EDuration="2.622736514s" totalImagesPullingTime="350.501808ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-13 22:47:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 22:47:02.621787786 +0000 UTC m=+2017.869703136" watchObservedRunningTime="2026-07-13 22:47:02.622736514 +0000 UTC m=+2017.870651864" Jul 13 22:47:05.621186 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:47:05.621154 2583 generic.go:356] "Generic (PLEG): container finished" podID="3b6b3625-bfa7-4be3-9b53-15294800bd4b" containerID="cf932f47c2539acda802b6a97fb0738ff7d49664aad3e5655551b515e410f046" exitCode=0 Jul 13 22:47:05.621783 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:47:05.621233 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-455d3b-openshift-default-7b9767fd8c-wtg94" event={"ID":"3b6b3625-bfa7-4be3-9b53-15294800bd4b","Type":"ContainerDied","Data":"cf932f47c2539acda802b6a97fb0738ff7d49664aad3e5655551b515e410f046"} Jul 13 22:47:05.621783 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:47:05.621778 2583 scope.go:117] "RemoveContainer" containerID="cf932f47c2539acda802b6a97fb0738ff7d49664aad3e5655551b515e410f046" Jul 13 22:47:05.622734 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:47:05.622704 2583 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 13 22:47:05.622805 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:47:05.622772 2583 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 13 22:47:05.622841 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:47:05.622802 2583 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 13 22:47:06.627666 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:47:06.627621 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-455d3b-openshift-default-7b9767fd8c-wtg94" event={"ID":"3b6b3625-bfa7-4be3-9b53-15294800bd4b","Type":"ContainerStarted","Data":"031e1a9ff58148706b56509af8b872e0797afdf342f2388dfa3388668101d9f5"} Jul 13 22:47:06.809402 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:47:06.809364 2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-455d3b-openshift-default-7b9767fd8c-wtg94" Jul 13 22:47:06.810390 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:47:06.810363 2583 patch_prober.go:28] interesting pod/e2e-shared-a-455d3b-openshift-default-7b9767fd8c-wtg94 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.41:15021/healthz/ready\": dial tcp 10.133.0.41:15021: connect: connection refused" start-of-body= Jul 13 22:47:06.810533 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:47:06.810418 2583 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-455d3b-openshift-default-7b9767fd8c-wtg94" podUID="3b6b3625-bfa7-4be3-9b53-15294800bd4b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.41:15021/healthz/ready\": dial tcp 10.133.0.41:15021: connect: connection refused" Jul 13 22:47:07.556967 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:47:07.556908 2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-455d3b-openshift-default-7b9767fd8c-wtg94" Jul 13 22:47:07.809411 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:47:07.809321 2583 patch_prober.go:28] interesting pod/e2e-shared-a-455d3b-openshift-default-7b9767fd8c-wtg94 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.41:15021/healthz/ready\": dial tcp 10.133.0.41:15021: connect: connection refused" start-of-body= Jul 13 22:47:07.809411 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:47:07.809378 2583 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-455d3b-openshift-default-7b9767fd8c-wtg94" podUID="3b6b3625-bfa7-4be3-9b53-15294800bd4b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.41:15021/healthz/ready\": dial tcp 10.133.0.41:15021: connect: connection refused" Jul 13 22:47:08.615886 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:47:08.615852 2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-b37782-84fd897778-24bxc" Jul 13 22:47:08.809642 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:47:08.809601 2583 patch_prober.go:28] interesting pod/e2e-shared-a-455d3b-openshift-default-7b9767fd8c-wtg94 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.41:15021/healthz/ready\": dial tcp 10.133.0.41:15021: connect: connection refused" start-of-body= Jul 13 22:47:08.810027 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:47:08.809665 2583 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-455d3b-openshift-default-7b9767fd8c-wtg94" podUID="3b6b3625-bfa7-4be3-9b53-15294800bd4b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.41:15021/healthz/ready\": dial tcp 10.133.0.41:15021: connect: connection refused" Jul 13 22:47:09.809563 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:47:09.809526 2583 patch_prober.go:28] interesting pod/e2e-shared-a-455d3b-openshift-default-7b9767fd8c-wtg94 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.41:15021/healthz/ready\": dial tcp 10.133.0.41:15021: connect: connection refused" start-of-body= Jul 13 22:47:09.809772 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:47:09.809613 2583 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-455d3b-openshift-default-7b9767fd8c-wtg94" podUID="3b6b3625-bfa7-4be3-9b53-15294800bd4b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.41:15021/healthz/ready\": dial tcp 10.133.0.41:15021: connect: connection refused" Jul 13 22:47:10.812832 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:47:10.812799 2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-455d3b-openshift-default-7b9767fd8c-wtg94" Jul 13 22:47:10.813811 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:47:10.813793 2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-455d3b-openshift-default-7b9767fd8c-wtg94" Jul 13 22:47:14.674113 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:47:14.674078 2583 generic.go:356] "Generic (PLEG): container finished" podID="8d9ee87d-31e3-459a-9510-9e985b2359bb" containerID="47c0d5c976c45a8bab8d14651a5400ae83ee4bb9d5e8d4a59362aa1fa80ad960" exitCode=0 Jul 13 22:47:14.674559 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:47:14.674120 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-b37782-openshift-default-59c894cff-6xtjx" event={"ID":"8d9ee87d-31e3-459a-9510-9e985b2359bb","Type":"ContainerDied","Data":"47c0d5c976c45a8bab8d14651a5400ae83ee4bb9d5e8d4a59362aa1fa80ad960"} Jul 13 22:47:14.674559 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:47:14.674466 2583 scope.go:117] "RemoveContainer" containerID="47c0d5c976c45a8bab8d14651a5400ae83ee4bb9d5e8d4a59362aa1fa80ad960" Jul 13 22:47:14.675487 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:47:14.675455 2583 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 13 22:47:14.675602 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:47:14.675536 2583 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 13 22:47:14.675684 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:47:14.675602 2583 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 13 22:47:15.595643 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:47:15.595600 2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-b37782-openshift-default-59c894cff-6xtjx" Jul 13 22:47:15.680176 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:47:15.680137 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-b37782-openshift-default-59c894cff-6xtjx" event={"ID":"8d9ee87d-31e3-459a-9510-9e985b2359bb","Type":"ContainerStarted","Data":"6fa2ad7c5b01c6e1682b9a32b1d21b900416c19f4d403e1e860a263b5d6af15d"} Jul 13 22:47:15.945599 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:47:15.945267 2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-b37782-openshift-default-59c894cff-6xtjx" Jul 13 22:47:15.947335 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:47:15.947293 2583 patch_prober.go:28] interesting pod/e2e-shared-b-b37782-openshift-default-59c894cff-6xtjx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.43:15021/healthz/ready\": dial tcp 10.133.0.43:15021: connect: connection refused" start-of-body= Jul 13 22:47:15.947613 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:47:15.947557 2583 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-b37782-openshift-default-59c894cff-6xtjx" podUID="8d9ee87d-31e3-459a-9510-9e985b2359bb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.43:15021/healthz/ready\": dial tcp 10.133.0.43:15021: connect: connection refused" Jul 13 22:47:16.945783 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:47:16.945742 2583 patch_prober.go:28] interesting pod/e2e-shared-b-b37782-openshift-default-59c894cff-6xtjx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.43:15021/healthz/ready\": dial tcp 10.133.0.43:15021: connect: connection refused" start-of-body= Jul 13 22:47:16.946383 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:47:16.946349 2583 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-b37782-openshift-default-59c894cff-6xtjx" podUID="8d9ee87d-31e3-459a-9510-9e985b2359bb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.43:15021/healthz/ready\": dial tcp 10.133.0.43:15021: connect: connection refused" Jul 13 22:47:17.945972 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:47:17.945409 2583 patch_prober.go:28] interesting pod/e2e-shared-b-b37782-openshift-default-59c894cff-6xtjx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.43:15021/healthz/ready\": dial tcp 10.133.0.43:15021: connect: connection refused" start-of-body= Jul 13 22:47:17.945972 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:47:17.945490 2583 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-b37782-openshift-default-59c894cff-6xtjx" podUID="8d9ee87d-31e3-459a-9510-9e985b2359bb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.43:15021/healthz/ready\": dial tcp 10.133.0.43:15021: connect: connection refused" Jul 13 22:47:18.945493 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:47:18.945434 2583 patch_prober.go:28] interesting pod/e2e-shared-b-b37782-openshift-default-59c894cff-6xtjx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.43:15021/healthz/ready\": dial tcp 10.133.0.43:15021: connect: connection refused" start-of-body= Jul 13 22:47:18.946241 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:47:18.945806 2583 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-b37782-openshift-default-59c894cff-6xtjx" podUID="8d9ee87d-31e3-459a-9510-9e985b2359bb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.43:15021/healthz/ready\": dial tcp 10.133.0.43:15021: connect: connection refused" Jul 13 22:47:19.949328 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:47:19.949293 2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-b37782-openshift-default-59c894cff-6xtjx" Jul 13 22:47:19.949710 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:47:19.949678 2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-b37782-openshift-default-59c894cff-6xtjx" Jul 13 22:47:19.950393 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:47:19.950370 2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-b37782-openshift-default-59c894cff-6xtjx" Jul 13 22:50:10.732944 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:50:10.732910 2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-455d3b-7bb79b47cf-29n6r"] Jul 13 22:50:10.733491 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:50:10.733119 2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-455d3b-7bb79b47cf-29n6r" podUID="6558610a-5e68-4d35-977a-7992732426d8" containerName="maas-api" containerID="cri-o://09987f1c15253b6d3143083c3a45c1bf22bd0c34dada317acafe8a0a4fee0ccd" gracePeriod=30 Jul 13 22:50:10.974893 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:50:10.974866 2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-455d3b-7bb79b47cf-29n6r" Jul 13 22:50:11.017168 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:50:11.017074 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-tng45\" (UniqueName: \"kubernetes.io/projected/6558610a-5e68-4d35-977a-7992732426d8-kube-api-access-tng45\") pod \"6558610a-5e68-4d35-977a-7992732426d8\" (UID: \"6558610a-5e68-4d35-977a-7992732426d8\") " Jul 13 22:50:11.017168 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:50:11.017152 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6558610a-5e68-4d35-977a-7992732426d8-maas-api-tls\") pod \"6558610a-5e68-4d35-977a-7992732426d8\" (UID: \"6558610a-5e68-4d35-977a-7992732426d8\") " Jul 13 22:50:11.017356 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:50:11.017247 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6558610a-5e68-4d35-977a-7992732426d8-openshift-service-ca\") pod \"6558610a-5e68-4d35-977a-7992732426d8\" (UID: \"6558610a-5e68-4d35-977a-7992732426d8\") " Jul 13 22:50:11.017640 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:50:11.017611 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6558610a-5e68-4d35-977a-7992732426d8-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "6558610a-5e68-4d35-977a-7992732426d8" (UID: "6558610a-5e68-4d35-977a-7992732426d8"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 13 22:50:11.019319 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:50:11.019288 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6558610a-5e68-4d35-977a-7992732426d8-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "6558610a-5e68-4d35-977a-7992732426d8" (UID: "6558610a-5e68-4d35-977a-7992732426d8"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 13 22:50:11.019430 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:50:11.019366 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6558610a-5e68-4d35-977a-7992732426d8-kube-api-access-tng45" (OuterVolumeSpecName: "kube-api-access-tng45") pod "6558610a-5e68-4d35-977a-7992732426d8" (UID: "6558610a-5e68-4d35-977a-7992732426d8"). InnerVolumeSpecName "kube-api-access-tng45". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 13 22:50:11.118140 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:50:11.118104 2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tng45\" (UniqueName: \"kubernetes.io/projected/6558610a-5e68-4d35-977a-7992732426d8-kube-api-access-tng45\") on node \"ip-10-0-133-179.ec2.internal\" DevicePath \"\"" Jul 13 22:50:11.118140 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:50:11.118133 2583 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6558610a-5e68-4d35-977a-7992732426d8-maas-api-tls\") on node \"ip-10-0-133-179.ec2.internal\" DevicePath \"\"" Jul 13 22:50:11.118140 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:50:11.118142 2583 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6558610a-5e68-4d35-977a-7992732426d8-openshift-service-ca\") on node \"ip-10-0-133-179.ec2.internal\" DevicePath \"\"" Jul 13 22:50:11.437498 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:50:11.437459 2583 generic.go:356] "Generic (PLEG): container finished" podID="6558610a-5e68-4d35-977a-7992732426d8" containerID="09987f1c15253b6d3143083c3a45c1bf22bd0c34dada317acafe8a0a4fee0ccd" exitCode=0 Jul 13 22:50:11.437708 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:50:11.437511 2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-455d3b-7bb79b47cf-29n6r" Jul 13 22:50:11.437708 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:50:11.437547 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-455d3b-7bb79b47cf-29n6r" event={"ID":"6558610a-5e68-4d35-977a-7992732426d8","Type":"ContainerDied","Data":"09987f1c15253b6d3143083c3a45c1bf22bd0c34dada317acafe8a0a4fee0ccd"} Jul 13 22:50:11.437708 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:50:11.437625 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-455d3b-7bb79b47cf-29n6r" event={"ID":"6558610a-5e68-4d35-977a-7992732426d8","Type":"ContainerDied","Data":"667581107faae4140de9275f4c9d81c58dd4aed05d93e0d4a4c40dc1f2831155"} Jul 13 22:50:11.437708 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:50:11.437656 2583 scope.go:117] "RemoveContainer" containerID="09987f1c15253b6d3143083c3a45c1bf22bd0c34dada317acafe8a0a4fee0ccd" Jul 13 22:50:11.445679 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:50:11.445661 2583 scope.go:117] "RemoveContainer" containerID="09987f1c15253b6d3143083c3a45c1bf22bd0c34dada317acafe8a0a4fee0ccd" Jul 13 22:50:11.445928 ip-10-0-133-179 kubenswrapper[2583]: E0713 22:50:11.445907 2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"09987f1c15253b6d3143083c3a45c1bf22bd0c34dada317acafe8a0a4fee0ccd\": container with ID starting with 09987f1c15253b6d3143083c3a45c1bf22bd0c34dada317acafe8a0a4fee0ccd not found: ID does not exist" containerID="09987f1c15253b6d3143083c3a45c1bf22bd0c34dada317acafe8a0a4fee0ccd" Jul 13 22:50:11.445982 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:50:11.445937 2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"09987f1c15253b6d3143083c3a45c1bf22bd0c34dada317acafe8a0a4fee0ccd"} err="failed to get container status \"09987f1c15253b6d3143083c3a45c1bf22bd0c34dada317acafe8a0a4fee0ccd\": rpc error: code = NotFound desc = could not find container \"09987f1c15253b6d3143083c3a45c1bf22bd0c34dada317acafe8a0a4fee0ccd\": container with ID starting with 09987f1c15253b6d3143083c3a45c1bf22bd0c34dada317acafe8a0a4fee0ccd not found: ID does not exist" Jul 13 22:50:11.456728 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:50:11.456703 2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-455d3b-7bb79b47cf-29n6r"] Jul 13 22:50:11.464783 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:50:11.464763 2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-455d3b-7bb79b47cf-29n6r"] Jul 13 22:50:13.389950 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:50:13.389918 2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6558610a-5e68-4d35-977a-7992732426d8" path="/var/lib/kubelet/pods/6558610a-5e68-4d35-977a-7992732426d8/volumes" Jul 13 22:50:16.587171 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:50:16.587136 2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-455d3b-openshift-default-7b9767fd8c-wtg94"] Jul 13 22:50:16.587664 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:50:16.587413 2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-455d3b-openshift-default-7b9767fd8c-wtg94" podUID="3b6b3625-bfa7-4be3-9b53-15294800bd4b" containerName="istio-proxy" containerID="cri-o://031e1a9ff58148706b56509af8b872e0797afdf342f2388dfa3388668101d9f5" gracePeriod=30 Jul 13 22:50:17.696250 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:50:17.696219 2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-b37782-84fd897778-24bxc"] Jul 13 22:50:17.696676 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:50:17.696435 2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-b37782-84fd897778-24bxc" podUID="a01fedb6-f2c1-413b-96dd-50beeefeb079" containerName="maas-api" containerID="cri-o://4e59daf043e268abb4eec99c90eed2b07326f085e999b64101413a64b852b3d4" gracePeriod=30 Jul 13 22:50:17.937830 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:50:17.937797 2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-b37782-84fd897778-24bxc" Jul 13 22:50:17.977561 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:50:17.977489 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jb59q\" (UniqueName: \"kubernetes.io/projected/a01fedb6-f2c1-413b-96dd-50beeefeb079-kube-api-access-jb59q\") pod \"a01fedb6-f2c1-413b-96dd-50beeefeb079\" (UID: \"a01fedb6-f2c1-413b-96dd-50beeefeb079\") " Jul 13 22:50:17.977561 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:50:17.977563 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a01fedb6-f2c1-413b-96dd-50beeefeb079-maas-api-tls\") pod \"a01fedb6-f2c1-413b-96dd-50beeefeb079\" (UID: \"a01fedb6-f2c1-413b-96dd-50beeefeb079\") " Jul 13 22:50:17.977786 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:50:17.977617 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a01fedb6-f2c1-413b-96dd-50beeefeb079-openshift-service-ca\") pod \"a01fedb6-f2c1-413b-96dd-50beeefeb079\" (UID: \"a01fedb6-f2c1-413b-96dd-50beeefeb079\") " Jul 13 22:50:17.978055 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:50:17.978029 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a01fedb6-f2c1-413b-96dd-50beeefeb079-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "a01fedb6-f2c1-413b-96dd-50beeefeb079" (UID: "a01fedb6-f2c1-413b-96dd-50beeefeb079"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 13 22:50:17.979679 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:50:17.979641 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a01fedb6-f2c1-413b-96dd-50beeefeb079-kube-api-access-jb59q" (OuterVolumeSpecName: "kube-api-access-jb59q") pod "a01fedb6-f2c1-413b-96dd-50beeefeb079" (UID: "a01fedb6-f2c1-413b-96dd-50beeefeb079"). InnerVolumeSpecName "kube-api-access-jb59q". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 13 22:50:17.979777 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:50:17.979698 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a01fedb6-f2c1-413b-96dd-50beeefeb079-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "a01fedb6-f2c1-413b-96dd-50beeefeb079" (UID: "a01fedb6-f2c1-413b-96dd-50beeefeb079"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 13 22:50:18.078682 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:50:18.078643 2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jb59q\" (UniqueName: \"kubernetes.io/projected/a01fedb6-f2c1-413b-96dd-50beeefeb079-kube-api-access-jb59q\") on node \"ip-10-0-133-179.ec2.internal\" DevicePath \"\"" Jul 13 22:50:18.078682 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:50:18.078679 2583 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a01fedb6-f2c1-413b-96dd-50beeefeb079-maas-api-tls\") on node \"ip-10-0-133-179.ec2.internal\" DevicePath \"\"" Jul 13 22:50:18.078682 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:50:18.078689 2583 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a01fedb6-f2c1-413b-96dd-50beeefeb079-openshift-service-ca\") on node \"ip-10-0-133-179.ec2.internal\" DevicePath \"\"" Jul 13 22:50:18.469321 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:50:18.469283 2583 generic.go:356] "Generic (PLEG): container finished" podID="a01fedb6-f2c1-413b-96dd-50beeefeb079" containerID="4e59daf043e268abb4eec99c90eed2b07326f085e999b64101413a64b852b3d4" exitCode=0 Jul 13 22:50:18.469321 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:50:18.469319 2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-b37782-84fd897778-24bxc" Jul 13 22:50:18.469596 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:50:18.469356 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-b37782-84fd897778-24bxc" event={"ID":"a01fedb6-f2c1-413b-96dd-50beeefeb079","Type":"ContainerDied","Data":"4e59daf043e268abb4eec99c90eed2b07326f085e999b64101413a64b852b3d4"} Jul 13 22:50:18.469596 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:50:18.469422 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-b37782-84fd897778-24bxc" event={"ID":"a01fedb6-f2c1-413b-96dd-50beeefeb079","Type":"ContainerDied","Data":"35652982f9eb62ff427eec028a2849807e6f37600013de26a2e78b4c64d5f485"} Jul 13 22:50:18.469596 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:50:18.469462 2583 scope.go:117] "RemoveContainer" containerID="4e59daf043e268abb4eec99c90eed2b07326f085e999b64101413a64b852b3d4" Jul 13 22:50:18.478700 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:50:18.478555 2583 scope.go:117] "RemoveContainer" containerID="4e59daf043e268abb4eec99c90eed2b07326f085e999b64101413a64b852b3d4" Jul 13 22:50:18.478890 ip-10-0-133-179 kubenswrapper[2583]: E0713 22:50:18.478856 2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4e59daf043e268abb4eec99c90eed2b07326f085e999b64101413a64b852b3d4\": container with ID starting with 4e59daf043e268abb4eec99c90eed2b07326f085e999b64101413a64b852b3d4 not found: ID does not exist" containerID="4e59daf043e268abb4eec99c90eed2b07326f085e999b64101413a64b852b3d4" Jul 13 22:50:18.478979 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:50:18.478892 2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4e59daf043e268abb4eec99c90eed2b07326f085e999b64101413a64b852b3d4"} err="failed to get container status \"4e59daf043e268abb4eec99c90eed2b07326f085e999b64101413a64b852b3d4\": rpc error: code = NotFound desc = could not find container \"4e59daf043e268abb4eec99c90eed2b07326f085e999b64101413a64b852b3d4\": container with ID starting with 4e59daf043e268abb4eec99c90eed2b07326f085e999b64101413a64b852b3d4 not found: ID does not exist" Jul 13 22:50:18.508024 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:50:18.507997 2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-b37782-84fd897778-24bxc"] Jul 13 22:50:18.523988 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:50:18.523945 2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-b37782-84fd897778-24bxc"] Jul 13 22:50:19.389564 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:50:19.389517 2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a01fedb6-f2c1-413b-96dd-50beeefeb079" path="/var/lib/kubelet/pods/a01fedb6-f2c1-413b-96dd-50beeefeb079/volumes" Jul 13 22:50:21.844444 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:50:21.844418 2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-455d3b-openshift-default-7b9767fd8c-wtg94" Jul 13 22:50:21.911714 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:50:21.911610 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3b6b3625-bfa7-4be3-9b53-15294800bd4b-istiod-ca-cert\") pod \"3b6b3625-bfa7-4be3-9b53-15294800bd4b\" (UID: \"3b6b3625-bfa7-4be3-9b53-15294800bd4b\") " Jul 13 22:50:21.911714 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:50:21.911690 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3b6b3625-bfa7-4be3-9b53-15294800bd4b-workload-certs\") pod \"3b6b3625-bfa7-4be3-9b53-15294800bd4b\" (UID: \"3b6b3625-bfa7-4be3-9b53-15294800bd4b\") " Jul 13 22:50:21.911932 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:50:21.911717 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5t626\" (UniqueName: \"kubernetes.io/projected/3b6b3625-bfa7-4be3-9b53-15294800bd4b-kube-api-access-5t626\") pod \"3b6b3625-bfa7-4be3-9b53-15294800bd4b\" (UID: \"3b6b3625-bfa7-4be3-9b53-15294800bd4b\") " Jul 13 22:50:21.911932 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:50:21.911757 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3b6b3625-bfa7-4be3-9b53-15294800bd4b-istio-envoy\") pod \"3b6b3625-bfa7-4be3-9b53-15294800bd4b\" (UID: \"3b6b3625-bfa7-4be3-9b53-15294800bd4b\") " Jul 13 22:50:21.911932 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:50:21.911784 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3b6b3625-bfa7-4be3-9b53-15294800bd4b-workload-socket\") pod \"3b6b3625-bfa7-4be3-9b53-15294800bd4b\" (UID: \"3b6b3625-bfa7-4be3-9b53-15294800bd4b\") " Jul 13 22:50:21.911932 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:50:21.911811 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3b6b3625-bfa7-4be3-9b53-15294800bd4b-istio-token\") pod \"3b6b3625-bfa7-4be3-9b53-15294800bd4b\" (UID: \"3b6b3625-bfa7-4be3-9b53-15294800bd4b\") " Jul 13 22:50:21.911932 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:50:21.911849 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3b6b3625-bfa7-4be3-9b53-15294800bd4b-istio-podinfo\") pod \"3b6b3625-bfa7-4be3-9b53-15294800bd4b\" (UID: \"3b6b3625-bfa7-4be3-9b53-15294800bd4b\") " Jul 13 22:50:21.911932 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:50:21.911897 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3b6b3625-bfa7-4be3-9b53-15294800bd4b-istio-data\") pod \"3b6b3625-bfa7-4be3-9b53-15294800bd4b\" (UID: \"3b6b3625-bfa7-4be3-9b53-15294800bd4b\") " Jul 13 22:50:21.912229 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:50:21.911971 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3b6b3625-bfa7-4be3-9b53-15294800bd4b-credential-socket\") pod \"3b6b3625-bfa7-4be3-9b53-15294800bd4b\" (UID: \"3b6b3625-bfa7-4be3-9b53-15294800bd4b\") " Jul 13 22:50:21.912229 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:50:21.912058 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3b6b3625-bfa7-4be3-9b53-15294800bd4b-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "3b6b3625-bfa7-4be3-9b53-15294800bd4b" (UID: "3b6b3625-bfa7-4be3-9b53-15294800bd4b"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 13 22:50:21.912229 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:50:21.912082 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3b6b3625-bfa7-4be3-9b53-15294800bd4b-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "3b6b3625-bfa7-4be3-9b53-15294800bd4b" (UID: "3b6b3625-bfa7-4be3-9b53-15294800bd4b"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 13 22:50:21.912229 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:50:21.912173 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3b6b3625-bfa7-4be3-9b53-15294800bd4b-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "3b6b3625-bfa7-4be3-9b53-15294800bd4b" (UID: "3b6b3625-bfa7-4be3-9b53-15294800bd4b"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 13 22:50:21.912418 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:50:21.912237 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3b6b3625-bfa7-4be3-9b53-15294800bd4b-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "3b6b3625-bfa7-4be3-9b53-15294800bd4b" (UID: "3b6b3625-bfa7-4be3-9b53-15294800bd4b"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 13 22:50:21.912418 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:50:21.912314 2583 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3b6b3625-bfa7-4be3-9b53-15294800bd4b-credential-socket\") on node \"ip-10-0-133-179.ec2.internal\" DevicePath \"\"" Jul 13 22:50:21.912418 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:50:21.912335 2583 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3b6b3625-bfa7-4be3-9b53-15294800bd4b-istiod-ca-cert\") on node \"ip-10-0-133-179.ec2.internal\" DevicePath \"\"" Jul 13 22:50:21.912418 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:50:21.912350 2583 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3b6b3625-bfa7-4be3-9b53-15294800bd4b-workload-certs\") on node \"ip-10-0-133-179.ec2.internal\" DevicePath \"\"" Jul 13 22:50:21.912418 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:50:21.912364 2583 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3b6b3625-bfa7-4be3-9b53-15294800bd4b-workload-socket\") on node \"ip-10-0-133-179.ec2.internal\" DevicePath \"\"" Jul 13 22:50:21.912884 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:50:21.912863 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3b6b3625-bfa7-4be3-9b53-15294800bd4b-istio-data" (OuterVolumeSpecName: "istio-data") pod "3b6b3625-bfa7-4be3-9b53-15294800bd4b" (UID: "3b6b3625-bfa7-4be3-9b53-15294800bd4b"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 13 22:50:21.913988 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:50:21.913962 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3b6b3625-bfa7-4be3-9b53-15294800bd4b-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "3b6b3625-bfa7-4be3-9b53-15294800bd4b" (UID: "3b6b3625-bfa7-4be3-9b53-15294800bd4b"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 13 22:50:21.914118 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:50:21.914097 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/3b6b3625-bfa7-4be3-9b53-15294800bd4b-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "3b6b3625-bfa7-4be3-9b53-15294800bd4b" (UID: "3b6b3625-bfa7-4be3-9b53-15294800bd4b"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 13 22:50:21.914418 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:50:21.914390 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3b6b3625-bfa7-4be3-9b53-15294800bd4b-kube-api-access-5t626" (OuterVolumeSpecName: "kube-api-access-5t626") pod "3b6b3625-bfa7-4be3-9b53-15294800bd4b" (UID: "3b6b3625-bfa7-4be3-9b53-15294800bd4b"). InnerVolumeSpecName "kube-api-access-5t626". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 13 22:50:21.914473 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:50:21.914396 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3b6b3625-bfa7-4be3-9b53-15294800bd4b-istio-token" (OuterVolumeSpecName: "istio-token") pod "3b6b3625-bfa7-4be3-9b53-15294800bd4b" (UID: "3b6b3625-bfa7-4be3-9b53-15294800bd4b"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 13 22:50:22.013781 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:50:22.013738 2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5t626\" (UniqueName: \"kubernetes.io/projected/3b6b3625-bfa7-4be3-9b53-15294800bd4b-kube-api-access-5t626\") on node \"ip-10-0-133-179.ec2.internal\" DevicePath \"\"" Jul 13 22:50:22.013781 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:50:22.013775 2583 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3b6b3625-bfa7-4be3-9b53-15294800bd4b-istio-envoy\") on node \"ip-10-0-133-179.ec2.internal\" DevicePath \"\"" Jul 13 22:50:22.013781 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:50:22.013787 2583 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3b6b3625-bfa7-4be3-9b53-15294800bd4b-istio-token\") on node \"ip-10-0-133-179.ec2.internal\" DevicePath \"\"" Jul 13 22:50:22.014030 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:50:22.013798 2583 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3b6b3625-bfa7-4be3-9b53-15294800bd4b-istio-podinfo\") on node \"ip-10-0-133-179.ec2.internal\" DevicePath \"\"" Jul 13 22:50:22.014030 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:50:22.013809 2583 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3b6b3625-bfa7-4be3-9b53-15294800bd4b-istio-data\") on node \"ip-10-0-133-179.ec2.internal\" DevicePath \"\"" Jul 13 22:50:22.488613 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:50:22.488555 2583 generic.go:356] "Generic (PLEG): container finished" podID="3b6b3625-bfa7-4be3-9b53-15294800bd4b" containerID="031e1a9ff58148706b56509af8b872e0797afdf342f2388dfa3388668101d9f5" exitCode=0 Jul 13 22:50:22.488613 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:50:22.488600 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-455d3b-openshift-default-7b9767fd8c-wtg94" event={"ID":"3b6b3625-bfa7-4be3-9b53-15294800bd4b","Type":"ContainerDied","Data":"031e1a9ff58148706b56509af8b872e0797afdf342f2388dfa3388668101d9f5"} Jul 13 22:50:22.488835 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:50:22.488625 2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-455d3b-openshift-default-7b9767fd8c-wtg94" Jul 13 22:50:22.488835 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:50:22.488660 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-455d3b-openshift-default-7b9767fd8c-wtg94" event={"ID":"3b6b3625-bfa7-4be3-9b53-15294800bd4b","Type":"ContainerDied","Data":"c99a4b2e37a4e7cb3592794e10848cc21d16ffcce108750fdc7f2544ee3998af"} Jul 13 22:50:22.488835 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:50:22.488684 2583 scope.go:117] "RemoveContainer" containerID="031e1a9ff58148706b56509af8b872e0797afdf342f2388dfa3388668101d9f5" Jul 13 22:50:22.498040 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:50:22.498016 2583 scope.go:117] "RemoveContainer" containerID="cf932f47c2539acda802b6a97fb0738ff7d49664aad3e5655551b515e410f046" Jul 13 22:50:22.506236 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:50:22.506218 2583 scope.go:117] "RemoveContainer" containerID="031e1a9ff58148706b56509af8b872e0797afdf342f2388dfa3388668101d9f5" Jul 13 22:50:22.506548 ip-10-0-133-179 kubenswrapper[2583]: E0713 22:50:22.506512 2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"031e1a9ff58148706b56509af8b872e0797afdf342f2388dfa3388668101d9f5\": container with ID starting with 031e1a9ff58148706b56509af8b872e0797afdf342f2388dfa3388668101d9f5 not found: ID does not exist" containerID="031e1a9ff58148706b56509af8b872e0797afdf342f2388dfa3388668101d9f5" Jul 13 22:50:22.506681 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:50:22.506558 2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"031e1a9ff58148706b56509af8b872e0797afdf342f2388dfa3388668101d9f5"} err="failed to get container status \"031e1a9ff58148706b56509af8b872e0797afdf342f2388dfa3388668101d9f5\": rpc error: code = NotFound desc = could not find container \"031e1a9ff58148706b56509af8b872e0797afdf342f2388dfa3388668101d9f5\": container with ID starting with 031e1a9ff58148706b56509af8b872e0797afdf342f2388dfa3388668101d9f5 not found: ID does not exist" Jul 13 22:50:22.506681 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:50:22.506598 2583 scope.go:117] "RemoveContainer" containerID="cf932f47c2539acda802b6a97fb0738ff7d49664aad3e5655551b515e410f046" Jul 13 22:50:22.507165 ip-10-0-133-179 kubenswrapper[2583]: E0713 22:50:22.507058 2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cf932f47c2539acda802b6a97fb0738ff7d49664aad3e5655551b515e410f046\": container with ID starting with cf932f47c2539acda802b6a97fb0738ff7d49664aad3e5655551b515e410f046 not found: ID does not exist" containerID="cf932f47c2539acda802b6a97fb0738ff7d49664aad3e5655551b515e410f046" Jul 13 22:50:22.507165 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:50:22.507109 2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cf932f47c2539acda802b6a97fb0738ff7d49664aad3e5655551b515e410f046"} err="failed to get container status \"cf932f47c2539acda802b6a97fb0738ff7d49664aad3e5655551b515e410f046\": rpc error: code = NotFound desc = could not find container \"cf932f47c2539acda802b6a97fb0738ff7d49664aad3e5655551b515e410f046\": container with ID starting with cf932f47c2539acda802b6a97fb0738ff7d49664aad3e5655551b515e410f046 not found: ID does not exist" Jul 13 22:50:22.508361 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:50:22.508338 2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-455d3b-openshift-default-7b9767fd8c-wtg94"] Jul 13 22:50:22.511186 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:50:22.511163 2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-455d3b-openshift-default-7b9767fd8c-wtg94"] Jul 13 22:50:23.391537 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:50:23.391501 2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3b6b3625-bfa7-4be3-9b53-15294800bd4b" path="/var/lib/kubelet/pods/3b6b3625-bfa7-4be3-9b53-15294800bd4b/volumes" Jul 13 22:50:23.391933 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:50:23.391900 2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-b37782-openshift-default-59c894cff-6xtjx"] Jul 13 22:50:23.392167 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:50:23.392141 2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-b37782-openshift-default-59c894cff-6xtjx" podUID="8d9ee87d-31e3-459a-9510-9e985b2359bb" containerName="istio-proxy" containerID="cri-o://6fa2ad7c5b01c6e1682b9a32b1d21b900416c19f4d403e1e860a263b5d6af15d" gracePeriod=30 Jul 13 22:50:28.520456 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:50:28.520424 2583 generic.go:356] "Generic (PLEG): container finished" podID="8d9ee87d-31e3-459a-9510-9e985b2359bb" containerID="6fa2ad7c5b01c6e1682b9a32b1d21b900416c19f4d403e1e860a263b5d6af15d" exitCode=0 Jul 13 22:50:28.520896 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:50:28.520498 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-b37782-openshift-default-59c894cff-6xtjx" event={"ID":"8d9ee87d-31e3-459a-9510-9e985b2359bb","Type":"ContainerDied","Data":"6fa2ad7c5b01c6e1682b9a32b1d21b900416c19f4d403e1e860a263b5d6af15d"} Jul 13 22:50:28.520896 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:50:28.520599 2583 scope.go:117] "RemoveContainer" containerID="47c0d5c976c45a8bab8d14651a5400ae83ee4bb9d5e8d4a59362aa1fa80ad960" Jul 13 22:50:28.641665 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:50:28.641642 2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-b37782-openshift-default-59c894cff-6xtjx" Jul 13 22:50:28.777458 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:50:28.777425 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8d9ee87d-31e3-459a-9510-9e985b2359bb-istio-envoy\") pod \"8d9ee87d-31e3-459a-9510-9e985b2359bb\" (UID: \"8d9ee87d-31e3-459a-9510-9e985b2359bb\") " Jul 13 22:50:28.777458 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:50:28.777467 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-fvmvc\" (UniqueName: \"kubernetes.io/projected/8d9ee87d-31e3-459a-9510-9e985b2359bb-kube-api-access-fvmvc\") pod \"8d9ee87d-31e3-459a-9510-9e985b2359bb\" (UID: \"8d9ee87d-31e3-459a-9510-9e985b2359bb\") " Jul 13 22:50:28.777748 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:50:28.777493 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8d9ee87d-31e3-459a-9510-9e985b2359bb-credential-socket\") pod \"8d9ee87d-31e3-459a-9510-9e985b2359bb\" (UID: \"8d9ee87d-31e3-459a-9510-9e985b2359bb\") " Jul 13 22:50:28.777748 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:50:28.777541 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8d9ee87d-31e3-459a-9510-9e985b2359bb-workload-certs\") pod \"8d9ee87d-31e3-459a-9510-9e985b2359bb\" (UID: \"8d9ee87d-31e3-459a-9510-9e985b2359bb\") " Jul 13 22:50:28.777748 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:50:28.777587 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8d9ee87d-31e3-459a-9510-9e985b2359bb-istio-token\") pod \"8d9ee87d-31e3-459a-9510-9e985b2359bb\" (UID: \"8d9ee87d-31e3-459a-9510-9e985b2359bb\") " Jul 13 22:50:28.777748 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:50:28.777613 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8d9ee87d-31e3-459a-9510-9e985b2359bb-istio-podinfo\") pod \"8d9ee87d-31e3-459a-9510-9e985b2359bb\" (UID: \"8d9ee87d-31e3-459a-9510-9e985b2359bb\") " Jul 13 22:50:28.777748 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:50:28.777643 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8d9ee87d-31e3-459a-9510-9e985b2359bb-istio-data\") pod \"8d9ee87d-31e3-459a-9510-9e985b2359bb\" (UID: \"8d9ee87d-31e3-459a-9510-9e985b2359bb\") " Jul 13 22:50:28.777748 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:50:28.777696 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8d9ee87d-31e3-459a-9510-9e985b2359bb-workload-socket\") pod \"8d9ee87d-31e3-459a-9510-9e985b2359bb\" (UID: \"8d9ee87d-31e3-459a-9510-9e985b2359bb\") " Jul 13 22:50:28.778049 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:50:28.777880 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8d9ee87d-31e3-459a-9510-9e985b2359bb-istiod-ca-cert\") pod \"8d9ee87d-31e3-459a-9510-9e985b2359bb\" (UID: \"8d9ee87d-31e3-459a-9510-9e985b2359bb\") " Jul 13 22:50:28.778049 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:50:28.777923 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8d9ee87d-31e3-459a-9510-9e985b2359bb-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "8d9ee87d-31e3-459a-9510-9e985b2359bb" (UID: "8d9ee87d-31e3-459a-9510-9e985b2359bb"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 13 22:50:28.778049 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:50:28.777941 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8d9ee87d-31e3-459a-9510-9e985b2359bb-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "8d9ee87d-31e3-459a-9510-9e985b2359bb" (UID: "8d9ee87d-31e3-459a-9510-9e985b2359bb"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 13 22:50:28.778222 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:50:28.778129 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8d9ee87d-31e3-459a-9510-9e985b2359bb-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "8d9ee87d-31e3-459a-9510-9e985b2359bb" (UID: "8d9ee87d-31e3-459a-9510-9e985b2359bb"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 13 22:50:28.778367 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:50:28.778341 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8d9ee87d-31e3-459a-9510-9e985b2359bb-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "8d9ee87d-31e3-459a-9510-9e985b2359bb" (UID: "8d9ee87d-31e3-459a-9510-9e985b2359bb"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 13 22:50:28.778634 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:50:28.778616 2583 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8d9ee87d-31e3-459a-9510-9e985b2359bb-istiod-ca-cert\") on node \"ip-10-0-133-179.ec2.internal\" DevicePath \"\"" Jul 13 22:50:28.778759 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:50:28.778637 2583 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8d9ee87d-31e3-459a-9510-9e985b2359bb-credential-socket\") on node \"ip-10-0-133-179.ec2.internal\" DevicePath \"\"" Jul 13 22:50:28.778759 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:50:28.778650 2583 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8d9ee87d-31e3-459a-9510-9e985b2359bb-workload-certs\") on node \"ip-10-0-133-179.ec2.internal\" DevicePath \"\"" Jul 13 22:50:28.778759 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:50:28.778664 2583 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8d9ee87d-31e3-459a-9510-9e985b2359bb-workload-socket\") on node \"ip-10-0-133-179.ec2.internal\" DevicePath \"\"" Jul 13 22:50:28.778923 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:50:28.778763 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8d9ee87d-31e3-459a-9510-9e985b2359bb-istio-data" (OuterVolumeSpecName: "istio-data") pod "8d9ee87d-31e3-459a-9510-9e985b2359bb" (UID: "8d9ee87d-31e3-459a-9510-9e985b2359bb"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 13 22:50:28.779884 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:50:28.779856 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8d9ee87d-31e3-459a-9510-9e985b2359bb-istio-token" (OuterVolumeSpecName: "istio-token") pod "8d9ee87d-31e3-459a-9510-9e985b2359bb" (UID: "8d9ee87d-31e3-459a-9510-9e985b2359bb"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 13 22:50:28.780000 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:50:28.779951 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/8d9ee87d-31e3-459a-9510-9e985b2359bb-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "8d9ee87d-31e3-459a-9510-9e985b2359bb" (UID: "8d9ee87d-31e3-459a-9510-9e985b2359bb"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 13 22:50:28.780138 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:50:28.780116 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8d9ee87d-31e3-459a-9510-9e985b2359bb-kube-api-access-fvmvc" (OuterVolumeSpecName: "kube-api-access-fvmvc") pod "8d9ee87d-31e3-459a-9510-9e985b2359bb" (UID: "8d9ee87d-31e3-459a-9510-9e985b2359bb"). InnerVolumeSpecName "kube-api-access-fvmvc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 13 22:50:28.780226 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:50:28.780206 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8d9ee87d-31e3-459a-9510-9e985b2359bb-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "8d9ee87d-31e3-459a-9510-9e985b2359bb" (UID: "8d9ee87d-31e3-459a-9510-9e985b2359bb"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 13 22:50:28.879846 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:50:28.879808 2583 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8d9ee87d-31e3-459a-9510-9e985b2359bb-istio-envoy\") on node \"ip-10-0-133-179.ec2.internal\" DevicePath \"\"" Jul 13 22:50:28.879846 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:50:28.879840 2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fvmvc\" (UniqueName: \"kubernetes.io/projected/8d9ee87d-31e3-459a-9510-9e985b2359bb-kube-api-access-fvmvc\") on node \"ip-10-0-133-179.ec2.internal\" DevicePath \"\"" Jul 13 22:50:28.879846 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:50:28.879854 2583 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8d9ee87d-31e3-459a-9510-9e985b2359bb-istio-token\") on node \"ip-10-0-133-179.ec2.internal\" DevicePath \"\"" Jul 13 22:50:28.880138 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:50:28.879862 2583 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8d9ee87d-31e3-459a-9510-9e985b2359bb-istio-podinfo\") on node \"ip-10-0-133-179.ec2.internal\" DevicePath \"\"" Jul 13 22:50:28.880138 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:50:28.879871 2583 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8d9ee87d-31e3-459a-9510-9e985b2359bb-istio-data\") on node \"ip-10-0-133-179.ec2.internal\" DevicePath \"\"" Jul 13 22:50:29.526136 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:50:29.526100 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-b37782-openshift-default-59c894cff-6xtjx" event={"ID":"8d9ee87d-31e3-459a-9510-9e985b2359bb","Type":"ContainerDied","Data":"b6710a251fe80bb1714e1d7c2db45451846151f5641dd353d96666866198b0c7"} Jul 13 22:50:29.526136 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:50:29.526119 2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-b37782-openshift-default-59c894cff-6xtjx" Jul 13 22:50:29.526642 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:50:29.526166 2583 scope.go:117] "RemoveContainer" containerID="6fa2ad7c5b01c6e1682b9a32b1d21b900416c19f4d403e1e860a263b5d6af15d" Jul 13 22:50:29.542211 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:50:29.542181 2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-b37782-openshift-default-59c894cff-6xtjx"] Jul 13 22:50:29.546468 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:50:29.546440 2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-b37782-openshift-default-59c894cff-6xtjx"] Jul 13 22:50:29.586128 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:50:29.586101 2583 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-d5bf5468b-bmrcj_9189e9b0-0cba-43f0-84a7-4a1024b285b0/copy-manifests/0.log" Jul 13 22:50:29.593817 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:50:29.593781 2583 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-d5bf5468b-bmrcj_9189e9b0-0cba-43f0-84a7-4a1024b285b0/manager/0.log" Jul 13 22:50:31.389389 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:50:31.389356 2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8d9ee87d-31e3-459a-9510-9e985b2359bb" path="/var/lib/kubelet/pods/8d9ee87d-31e3-459a-9510-9e985b2359bb/volumes" Jul 13 22:50:32.178065 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:50:32.178030 2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-7f964fb678lhs8d_318be4cb-7ac5-4039-a1be-f26e6f9f4d18/istio-proxy/0.log" Jul 13 22:50:32.303869 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:50:32.303832 2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-94bb8fbfd-28vcl_52bdd097-54ac-4137-aced-c5b4debbb854/discovery/0.log" Jul 13 22:50:32.781700 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:50:32.781674 2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-7964457696-49fdw_617e7821-66c5-4515-b29a-b0e3ed27655f/payload-pre-processing/0.log" Jul 13 22:50:33.483879 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:50:33.483854 2583 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-distinct-simulated-kserve-59975d69f9-5jd66_81dac3d5-64d0-4c88-9350-07e5b0fc18fc/main/0.log" Jul 13 22:50:33.826298 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:50:33.826258 2583 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_facebook-opt-125m-simulated-kserve-74b68447df-l8v2f_bf682d14-b0e6-484f-bad2-ac5790938edf/main/0.log" Jul 13 22:50:40.967751 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:50:40.967716 2583 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-9hl5q_d38c850d-d65b-4f99-ad24-5c1d443bff4b/global-pull-secret-syncer/0.log" Jul 13 22:50:41.185320 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:50:41.185284 2583 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-wgcj4_ad16aa2f-aede-4d47-82d6-0ab7dda663d4/konnectivity-agent/0.log" Jul 13 22:50:41.227964 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:50:41.227887 2583 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-133-179.ec2.internal_120f6b7b6960a904bf802202dca0166e/haproxy/0.log" Jul 13 22:50:48.598530 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:50:48.598493 2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-9fx68_d060f80c-835e-49e4-a012-e32fdbdb3890/node-exporter/0.log" Jul 13 22:50:48.616875 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:50:48.616846 2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-9fx68_d060f80c-835e-49e4-a012-e32fdbdb3890/kube-rbac-proxy/0.log" Jul 13 22:50:48.635722 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:50:48.635695 2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-9fx68_d060f80c-835e-49e4-a012-e32fdbdb3890/init-textfile/0.log" Jul 13 22:50:51.048250 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:50:51.048210 2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-g7vdh/perf-node-gather-daemonset-zhmdl"] Jul 13 22:50:51.048772 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:50:51.048660 2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8d9ee87d-31e3-459a-9510-9e985b2359bb" containerName="istio-proxy" Jul 13 22:50:51.048772 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:50:51.048679 2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="8d9ee87d-31e3-459a-9510-9e985b2359bb" containerName="istio-proxy" Jul 13 22:50:51.048772 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:50:51.048699 2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a01fedb6-f2c1-413b-96dd-50beeefeb079" containerName="maas-api" Jul 13 22:50:51.048772 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:50:51.048707 2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="a01fedb6-f2c1-413b-96dd-50beeefeb079" containerName="maas-api" Jul 13 22:50:51.048772 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:50:51.048721 2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3b6b3625-bfa7-4be3-9b53-15294800bd4b" containerName="istio-proxy" Jul 13 22:50:51.048772 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:50:51.048728 2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="3b6b3625-bfa7-4be3-9b53-15294800bd4b" containerName="istio-proxy" Jul 13 22:50:51.048772 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:50:51.048736 2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8d9ee87d-31e3-459a-9510-9e985b2359bb" containerName="istio-proxy" Jul 13 22:50:51.048772 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:50:51.048744 2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="8d9ee87d-31e3-459a-9510-9e985b2359bb" containerName="istio-proxy" Jul 13 22:50:51.048772 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:50:51.048755 2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3b6b3625-bfa7-4be3-9b53-15294800bd4b" containerName="istio-proxy" Jul 13 22:50:51.048772 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:50:51.048762 2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="3b6b3625-bfa7-4be3-9b53-15294800bd4b" containerName="istio-proxy" Jul 13 22:50:51.048772 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:50:51.048772 2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6558610a-5e68-4d35-977a-7992732426d8" containerName="maas-api" Jul 13 22:50:51.049292 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:50:51.048779 2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="6558610a-5e68-4d35-977a-7992732426d8" containerName="maas-api" Jul 13 22:50:51.049292 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:50:51.048867 2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="6558610a-5e68-4d35-977a-7992732426d8" containerName="maas-api" Jul 13 22:50:51.049292 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:50:51.048878 2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="a01fedb6-f2c1-413b-96dd-50beeefeb079" containerName="maas-api" Jul 13 22:50:51.049292 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:50:51.048890 2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="3b6b3625-bfa7-4be3-9b53-15294800bd4b" containerName="istio-proxy" Jul 13 22:50:51.049292 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:50:51.048899 2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="8d9ee87d-31e3-459a-9510-9e985b2359bb" containerName="istio-proxy" Jul 13 22:50:51.049292 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:50:51.048907 2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="8d9ee87d-31e3-459a-9510-9e985b2359bb" containerName="istio-proxy" Jul 13 22:50:51.049292 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:50:51.048918 2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="3b6b3625-bfa7-4be3-9b53-15294800bd4b" containerName="istio-proxy" Jul 13 22:50:51.052289 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:50:51.052266 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-g7vdh/perf-node-gather-daemonset-zhmdl" Jul 13 22:50:51.054421 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:50:51.054396 2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-g7vdh\"/\"kube-root-ca.crt\"" Jul 13 22:50:51.054558 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:50:51.054459 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-g7vdh\"/\"default-dockercfg-c6x79\"" Jul 13 22:50:51.055233 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:50:51.055218 2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-g7vdh\"/\"openshift-service-ca.crt\"" Jul 13 22:50:51.058189 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:50:51.058160 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h6fbl\" (UniqueName: \"kubernetes.io/projected/40cc5972-28ee-46ac-ae19-8fa282041d18-kube-api-access-h6fbl\") pod \"perf-node-gather-daemonset-zhmdl\" (UID: \"40cc5972-28ee-46ac-ae19-8fa282041d18\") " pod="openshift-must-gather-g7vdh/perf-node-gather-daemonset-zhmdl" Jul 13 22:50:51.058189 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:50:51.058161 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-g7vdh/perf-node-gather-daemonset-zhmdl"] Jul 13 22:50:51.058343 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:50:51.058218 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/40cc5972-28ee-46ac-ae19-8fa282041d18-lib-modules\") pod \"perf-node-gather-daemonset-zhmdl\" (UID: \"40cc5972-28ee-46ac-ae19-8fa282041d18\") " pod="openshift-must-gather-g7vdh/perf-node-gather-daemonset-zhmdl" Jul 13 22:50:51.058343 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:50:51.058301 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/40cc5972-28ee-46ac-ae19-8fa282041d18-sys\") pod \"perf-node-gather-daemonset-zhmdl\" (UID: \"40cc5972-28ee-46ac-ae19-8fa282041d18\") " pod="openshift-must-gather-g7vdh/perf-node-gather-daemonset-zhmdl" Jul 13 22:50:51.058343 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:50:51.058337 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/40cc5972-28ee-46ac-ae19-8fa282041d18-proc\") pod \"perf-node-gather-daemonset-zhmdl\" (UID: \"40cc5972-28ee-46ac-ae19-8fa282041d18\") " pod="openshift-must-gather-g7vdh/perf-node-gather-daemonset-zhmdl" Jul 13 22:50:51.058457 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:50:51.058382 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/40cc5972-28ee-46ac-ae19-8fa282041d18-podres\") pod \"perf-node-gather-daemonset-zhmdl\" (UID: \"40cc5972-28ee-46ac-ae19-8fa282041d18\") " pod="openshift-must-gather-g7vdh/perf-node-gather-daemonset-zhmdl" Jul 13 22:50:51.159778 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:50:51.159742 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h6fbl\" (UniqueName: \"kubernetes.io/projected/40cc5972-28ee-46ac-ae19-8fa282041d18-kube-api-access-h6fbl\") pod \"perf-node-gather-daemonset-zhmdl\" (UID: \"40cc5972-28ee-46ac-ae19-8fa282041d18\") " pod="openshift-must-gather-g7vdh/perf-node-gather-daemonset-zhmdl" Jul 13 22:50:51.159778 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:50:51.159782 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/40cc5972-28ee-46ac-ae19-8fa282041d18-lib-modules\") pod \"perf-node-gather-daemonset-zhmdl\" (UID: \"40cc5972-28ee-46ac-ae19-8fa282041d18\") " pod="openshift-must-gather-g7vdh/perf-node-gather-daemonset-zhmdl" Jul 13 22:50:51.160040 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:50:51.159808 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/40cc5972-28ee-46ac-ae19-8fa282041d18-sys\") pod \"perf-node-gather-daemonset-zhmdl\" (UID: \"40cc5972-28ee-46ac-ae19-8fa282041d18\") " pod="openshift-must-gather-g7vdh/perf-node-gather-daemonset-zhmdl" Jul 13 22:50:51.160040 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:50:51.159826 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/40cc5972-28ee-46ac-ae19-8fa282041d18-proc\") pod \"perf-node-gather-daemonset-zhmdl\" (UID: \"40cc5972-28ee-46ac-ae19-8fa282041d18\") " pod="openshift-must-gather-g7vdh/perf-node-gather-daemonset-zhmdl" Jul 13 22:50:51.160040 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:50:51.159851 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/40cc5972-28ee-46ac-ae19-8fa282041d18-podres\") pod \"perf-node-gather-daemonset-zhmdl\" (UID: \"40cc5972-28ee-46ac-ae19-8fa282041d18\") " pod="openshift-must-gather-g7vdh/perf-node-gather-daemonset-zhmdl" Jul 13 22:50:51.160040 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:50:51.159904 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/40cc5972-28ee-46ac-ae19-8fa282041d18-sys\") pod \"perf-node-gather-daemonset-zhmdl\" (UID: \"40cc5972-28ee-46ac-ae19-8fa282041d18\") " pod="openshift-must-gather-g7vdh/perf-node-gather-daemonset-zhmdl" Jul 13 22:50:51.160040 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:50:51.159950 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/40cc5972-28ee-46ac-ae19-8fa282041d18-lib-modules\") pod \"perf-node-gather-daemonset-zhmdl\" (UID: \"40cc5972-28ee-46ac-ae19-8fa282041d18\") " pod="openshift-must-gather-g7vdh/perf-node-gather-daemonset-zhmdl" Jul 13 22:50:51.160040 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:50:51.159972 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/40cc5972-28ee-46ac-ae19-8fa282041d18-proc\") pod \"perf-node-gather-daemonset-zhmdl\" (UID: \"40cc5972-28ee-46ac-ae19-8fa282041d18\") " pod="openshift-must-gather-g7vdh/perf-node-gather-daemonset-zhmdl" Jul 13 22:50:51.160040 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:50:51.159956 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/40cc5972-28ee-46ac-ae19-8fa282041d18-podres\") pod \"perf-node-gather-daemonset-zhmdl\" (UID: \"40cc5972-28ee-46ac-ae19-8fa282041d18\") " pod="openshift-must-gather-g7vdh/perf-node-gather-daemonset-zhmdl" Jul 13 22:50:51.168043 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:50:51.168009 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-h6fbl\" (UniqueName: \"kubernetes.io/projected/40cc5972-28ee-46ac-ae19-8fa282041d18-kube-api-access-h6fbl\") pod \"perf-node-gather-daemonset-zhmdl\" (UID: \"40cc5972-28ee-46ac-ae19-8fa282041d18\") " pod="openshift-must-gather-g7vdh/perf-node-gather-daemonset-zhmdl" Jul 13 22:50:51.363997 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:50:51.363898 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-g7vdh/perf-node-gather-daemonset-zhmdl" Jul 13 22:50:51.495418 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:50:51.495379 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-g7vdh/perf-node-gather-daemonset-zhmdl"] Jul 13 22:50:51.498671 ip-10-0-133-179 kubenswrapper[2583]: W0713 22:50:51.498642 2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod40cc5972_28ee_46ac_ae19_8fa282041d18.slice/crio-7b0f14b8f109f5e7526caf5f25e9a58b09d0770ccb5b34e5b4dfcb704c831dd8 WatchSource:0}: Error finding container 7b0f14b8f109f5e7526caf5f25e9a58b09d0770ccb5b34e5b4dfcb704c831dd8: Status 404 returned error can't find the container with id 7b0f14b8f109f5e7526caf5f25e9a58b09d0770ccb5b34e5b4dfcb704c831dd8 Jul 13 22:50:51.613347 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:50:51.613309 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-g7vdh/perf-node-gather-daemonset-zhmdl" event={"ID":"40cc5972-28ee-46ac-ae19-8fa282041d18","Type":"ContainerStarted","Data":"7b0f14b8f109f5e7526caf5f25e9a58b09d0770ccb5b34e5b4dfcb704c831dd8"} Jul 13 22:50:52.619520 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:50:52.619485 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-g7vdh/perf-node-gather-daemonset-zhmdl" event={"ID":"40cc5972-28ee-46ac-ae19-8fa282041d18","Type":"ContainerStarted","Data":"adac396918975f6c9c1a3360fcff8e3217557e3fdabf3dfe456a3c5008e35f0e"} Jul 13 22:50:52.619991 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:50:52.619622 2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-g7vdh/perf-node-gather-daemonset-zhmdl" Jul 13 22:50:52.635600 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:50:52.635531 2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-g7vdh/perf-node-gather-daemonset-zhmdl" podStartSLOduration=1.635513958 podStartE2EDuration="1.635513958s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-13 22:50:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 22:50:52.633973836 +0000 UTC m=+2247.881889198" watchObservedRunningTime="2026-07-13 22:50:52.635513958 +0000 UTC m=+2247.883429309" Jul 13 22:50:53.439827 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:50:53.439789 2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-w4tml_e92b34db-2297-4fc3-baab-7babcc0a94ff/dns/0.log" Jul 13 22:50:53.457991 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:50:53.457957 2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-w4tml_e92b34db-2297-4fc3-baab-7babcc0a94ff/kube-rbac-proxy/0.log" Jul 13 22:50:53.523774 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:50:53.523744 2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-grjf8_e8e90a01-fd99-4a55-ba17-522966f4fabe/dns-node-resolver/0.log" Jul 13 22:50:54.057769 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:50:54.057733 2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_image-registry-f7c956786-wvmw5_0ebe20b4-d6b9-404a-ac6f-a5c7894148cc/registry/0.log" Jul 13 22:50:54.124606 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:50:54.124558 2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-r26w6_54e8db97-3d72-4aa1-80bb-e56f10c7e474/node-ca/0.log" Jul 13 22:50:55.148972 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:50:55.148943 2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-7f964fb678lhs8d_318be4cb-7ac5-4039-a1be-f26e6f9f4d18/istio-proxy/0.log" Jul 13 22:50:55.316415 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:50:55.316377 2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-94bb8fbfd-28vcl_52bdd097-54ac-4137-aced-c5b4debbb854/discovery/0.log" Jul 13 22:50:55.497229 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:50:55.497141 2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-7964457696-49fdw_617e7821-66c5-4515-b29a-b0e3ed27655f/payload-pre-processing/0.log" Jul 13 22:50:56.319550 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:50:56.319522 2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-sbw7w_f8f9da95-14d8-4f4f-8142-0a2f347ba336/serve-healthcheck-canary/0.log" Jul 13 22:50:56.795957 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:50:56.795931 2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-dxppg_479a52b0-c5da-471f-a153-02d4155abe62/kube-rbac-proxy/0.log" Jul 13 22:50:56.815431 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:50:56.815404 2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-dxppg_479a52b0-c5da-471f-a153-02d4155abe62/exporter/0.log" Jul 13 22:50:56.838229 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:50:56.838185 2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-dxppg_479a52b0-c5da-471f-a153-02d4155abe62/extractor/0.log" Jul 13 22:50:58.633948 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:50:58.633918 2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-g7vdh/perf-node-gather-daemonset-zhmdl" Jul 13 22:50:59.510264 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:50:59.510230 2583 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-d5bf5468b-bmrcj_9189e9b0-0cba-43f0-84a7-4a1024b285b0/manager/0.log" Jul 13 22:50:59.530620 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:50:59.530595 2583 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-d5bf5468b-bmrcj_9189e9b0-0cba-43f0-84a7-4a1024b285b0/copy-manifests/0.log" Jul 13 22:51:00.989129 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:51:00.989103 2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-lws-operator_lws-controller-manager-84865b6cc-mqgj2_b42903c8-449f-4b60-8749-79e3ffeb03b7/manager/0.log" Jul 13 22:51:01.262258 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:51:01.262156 2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-7f964fb678lhs8d_318be4cb-7ac5-4039-a1be-f26e6f9f4d18/istio-proxy/0.log" Jul 13 22:51:01.423074 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:51:01.423042 2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-94bb8fbfd-28vcl_52bdd097-54ac-4137-aced-c5b4debbb854/discovery/0.log" Jul 13 22:51:01.573282 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:51:01.573244 2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-7964457696-49fdw_617e7821-66c5-4515-b29a-b0e3ed27655f/payload-pre-processing/0.log" Jul 13 22:51:02.769280 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:51:02.769189 2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-7f964fb678lhs8d_318be4cb-7ac5-4039-a1be-f26e6f9f4d18/istio-proxy/0.log" Jul 13 22:51:02.941920 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:51:02.941884 2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-94bb8fbfd-28vcl_52bdd097-54ac-4137-aced-c5b4debbb854/discovery/0.log" Jul 13 22:51:03.090022 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:51:03.089936 2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-7964457696-49fdw_617e7821-66c5-4515-b29a-b0e3ed27655f/payload-pre-processing/0.log" Jul 13 22:51:06.021144 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:51:06.021114 2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-7f59b54f6c-sfztp_ad1974c1-a1f3-41bc-aed1-133b74495c7e/migrator/0.log" Jul 13 22:51:06.040393 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:51:06.040366 2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-7f59b54f6c-sfztp_ad1974c1-a1f3-41bc-aed1-133b74495c7e/graceful-termination/0.log" Jul 13 22:51:07.413788 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:51:07.413753 2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-2jc7k_01c7e40b-ef15-4cf6-b5e7-e7129825d668/kube-multus/0.log" Jul 13 22:51:07.752256 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:51:07.752158 2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-nbdsh_03f58cdb-4d8b-41a9-aa82-dff50f158f0c/kube-multus-additional-cni-plugins/0.log" Jul 13 22:51:07.774529 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:51:07.774503 2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-nbdsh_03f58cdb-4d8b-41a9-aa82-dff50f158f0c/egress-router-binary-copy/0.log" Jul 13 22:51:07.796461 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:51:07.796433 2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-nbdsh_03f58cdb-4d8b-41a9-aa82-dff50f158f0c/cni-plugins/0.log" Jul 13 22:51:07.817753 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:51:07.817726 2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-nbdsh_03f58cdb-4d8b-41a9-aa82-dff50f158f0c/bond-cni-plugin/0.log" Jul 13 22:51:07.838906 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:51:07.838876 2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-nbdsh_03f58cdb-4d8b-41a9-aa82-dff50f158f0c/routeoverride-cni/0.log" Jul 13 22:51:07.859318 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:51:07.859289 2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-nbdsh_03f58cdb-4d8b-41a9-aa82-dff50f158f0c/whereabouts-cni-bincopy/0.log" Jul 13 22:51:07.879410 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:51:07.879378 2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-nbdsh_03f58cdb-4d8b-41a9-aa82-dff50f158f0c/whereabouts-cni/0.log" Jul 13 22:51:07.941394 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:51:07.941363 2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-26w5h_85265522-e114-4c04-b595-a26fcae86884/network-metrics-daemon/0.log" Jul 13 22:51:07.958743 ip-10-0-133-179 kubenswrapper[2583]: I0713 22:51:07.958694 2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-26w5h_85265522-e114-4c04-b595-a26fcae86884/kube-rbac-proxy/0.log"