Sep 10 13:17:17.923416 ip-10-0-139-174 systemd[1]: Starting Kubernetes Kubelet... Sep 10 13:17:18.326967 ip-10-0-139-174 kubenswrapper[2557]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Sep 10 13:17:18.326967 ip-10-0-139-174 kubenswrapper[2557]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version. Sep 10 13:17:18.326967 ip-10-0-139-174 kubenswrapper[2557]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Sep 10 13:17:18.326967 ip-10-0-139-174 kubenswrapper[2557]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI. Sep 10 13:17:18.326967 ip-10-0-139-174 kubenswrapper[2557]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Sep 10 13:17:18.328620 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:17:18.328549 2557 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime" Sep 10 13:17:18.333383 ip-10-0-139-174 kubenswrapper[2557]: W0910 13:17:18.333367 2557 feature_gate.go:328] unrecognized feature gate: NewOLM Sep 10 13:17:18.333383 ip-10-0-139-174 kubenswrapper[2557]: W0910 13:17:18.333382 2557 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Sep 10 13:17:18.333448 ip-10-0-139-174 kubenswrapper[2557]: W0910 13:17:18.333386 2557 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Sep 10 13:17:18.333448 ip-10-0-139-174 kubenswrapper[2557]: W0910 13:17:18.333389 2557 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Sep 10 13:17:18.333448 ip-10-0-139-174 kubenswrapper[2557]: W0910 13:17:18.333392 2557 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Sep 10 13:17:18.333448 ip-10-0-139-174 kubenswrapper[2557]: W0910 13:17:18.333395 2557 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Sep 10 13:17:18.333448 ip-10-0-139-174 kubenswrapper[2557]: W0910 13:17:18.333397 2557 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Sep 10 13:17:18.333448 ip-10-0-139-174 kubenswrapper[2557]: W0910 13:17:18.333400 2557 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Sep 10 13:17:18.333448 ip-10-0-139-174 kubenswrapper[2557]: W0910 13:17:18.333402 2557 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Sep 10 13:17:18.333448 ip-10-0-139-174 kubenswrapper[2557]: W0910 13:17:18.333405 2557 feature_gate.go:328] unrecognized feature gate: KMSEncryption Sep 10 13:17:18.333448 ip-10-0-139-174 kubenswrapper[2557]: W0910 13:17:18.333409 2557 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Sep 10 13:17:18.333448 ip-10-0-139-174 kubenswrapper[2557]: W0910 13:17:18.333411 2557 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Sep 10 13:17:18.333448 ip-10-0-139-174 kubenswrapper[2557]: W0910 13:17:18.333414 2557 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Sep 10 13:17:18.333448 ip-10-0-139-174 kubenswrapper[2557]: W0910 13:17:18.333417 2557 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Sep 10 13:17:18.333448 ip-10-0-139-174 kubenswrapper[2557]: W0910 13:17:18.333419 2557 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Sep 10 13:17:18.333448 ip-10-0-139-174 kubenswrapper[2557]: W0910 13:17:18.333422 2557 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Sep 10 13:17:18.333448 ip-10-0-139-174 kubenswrapper[2557]: W0910 13:17:18.333424 2557 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Sep 10 13:17:18.333448 ip-10-0-139-174 kubenswrapper[2557]: W0910 13:17:18.333427 2557 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Sep 10 13:17:18.333448 ip-10-0-139-174 kubenswrapper[2557]: W0910 13:17:18.333429 2557 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Sep 10 13:17:18.333448 ip-10-0-139-174 kubenswrapper[2557]: W0910 13:17:18.333432 2557 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Sep 10 13:17:18.333448 ip-10-0-139-174 kubenswrapper[2557]: W0910 13:17:18.333434 2557 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Sep 10 13:17:18.333448 ip-10-0-139-174 kubenswrapper[2557]: W0910 13:17:18.333436 2557 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Sep 10 13:17:18.333950 ip-10-0-139-174 kubenswrapper[2557]: W0910 13:17:18.333439 2557 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Sep 10 13:17:18.333950 ip-10-0-139-174 kubenswrapper[2557]: W0910 13:17:18.333447 2557 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Sep 10 13:17:18.333950 ip-10-0-139-174 kubenswrapper[2557]: W0910 13:17:18.333450 2557 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Sep 10 13:17:18.333950 ip-10-0-139-174 kubenswrapper[2557]: W0910 13:17:18.333454 2557 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Sep 10 13:17:18.333950 ip-10-0-139-174 kubenswrapper[2557]: W0910 13:17:18.333459 2557 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Sep 10 13:17:18.333950 ip-10-0-139-174 kubenswrapper[2557]: W0910 13:17:18.333463 2557 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Sep 10 13:17:18.333950 ip-10-0-139-174 kubenswrapper[2557]: W0910 13:17:18.333465 2557 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Sep 10 13:17:18.333950 ip-10-0-139-174 kubenswrapper[2557]: W0910 13:17:18.333468 2557 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Sep 10 13:17:18.333950 ip-10-0-139-174 kubenswrapper[2557]: W0910 13:17:18.333471 2557 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Sep 10 13:17:18.333950 ip-10-0-139-174 kubenswrapper[2557]: W0910 13:17:18.333473 2557 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Sep 10 13:17:18.333950 ip-10-0-139-174 kubenswrapper[2557]: W0910 13:17:18.333476 2557 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Sep 10 13:17:18.333950 ip-10-0-139-174 kubenswrapper[2557]: W0910 13:17:18.333478 2557 feature_gate.go:328] unrecognized feature gate: OSStreams Sep 10 13:17:18.333950 ip-10-0-139-174 kubenswrapper[2557]: W0910 13:17:18.333482 2557 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Sep 10 13:17:18.333950 ip-10-0-139-174 kubenswrapper[2557]: W0910 13:17:18.333486 2557 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Sep 10 13:17:18.333950 ip-10-0-139-174 kubenswrapper[2557]: W0910 13:17:18.333489 2557 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Sep 10 13:17:18.333950 ip-10-0-139-174 kubenswrapper[2557]: W0910 13:17:18.333492 2557 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Sep 10 13:17:18.333950 ip-10-0-139-174 kubenswrapper[2557]: W0910 13:17:18.333495 2557 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Sep 10 13:17:18.333950 ip-10-0-139-174 kubenswrapper[2557]: W0910 13:17:18.333498 2557 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Sep 10 13:17:18.334419 ip-10-0-139-174 kubenswrapper[2557]: W0910 13:17:18.333501 2557 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Sep 10 13:17:18.334419 ip-10-0-139-174 kubenswrapper[2557]: W0910 13:17:18.333504 2557 feature_gate.go:328] unrecognized feature gate: GatewayAPI Sep 10 13:17:18.334419 ip-10-0-139-174 kubenswrapper[2557]: W0910 13:17:18.333507 2557 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Sep 10 13:17:18.334419 ip-10-0-139-174 kubenswrapper[2557]: W0910 13:17:18.333509 2557 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Sep 10 13:17:18.334419 ip-10-0-139-174 kubenswrapper[2557]: W0910 13:17:18.333512 2557 feature_gate.go:328] unrecognized feature gate: Example Sep 10 13:17:18.334419 ip-10-0-139-174 kubenswrapper[2557]: W0910 13:17:18.333514 2557 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Sep 10 13:17:18.334419 ip-10-0-139-174 kubenswrapper[2557]: W0910 13:17:18.333517 2557 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Sep 10 13:17:18.334419 ip-10-0-139-174 kubenswrapper[2557]: W0910 13:17:18.333519 2557 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Sep 10 13:17:18.334419 ip-10-0-139-174 kubenswrapper[2557]: W0910 13:17:18.333521 2557 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Sep 10 13:17:18.334419 ip-10-0-139-174 kubenswrapper[2557]: W0910 13:17:18.333524 2557 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Sep 10 13:17:18.334419 ip-10-0-139-174 kubenswrapper[2557]: W0910 13:17:18.333526 2557 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Sep 10 13:17:18.334419 ip-10-0-139-174 kubenswrapper[2557]: W0910 13:17:18.333528 2557 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Sep 10 13:17:18.334419 ip-10-0-139-174 kubenswrapper[2557]: W0910 13:17:18.333531 2557 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Sep 10 13:17:18.334419 ip-10-0-139-174 kubenswrapper[2557]: W0910 13:17:18.333533 2557 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Sep 10 13:17:18.334419 ip-10-0-139-174 kubenswrapper[2557]: W0910 13:17:18.333536 2557 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Sep 10 13:17:18.334419 ip-10-0-139-174 kubenswrapper[2557]: W0910 13:17:18.333539 2557 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Sep 10 13:17:18.334419 ip-10-0-139-174 kubenswrapper[2557]: W0910 13:17:18.333541 2557 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Sep 10 13:17:18.334419 ip-10-0-139-174 kubenswrapper[2557]: W0910 13:17:18.333544 2557 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Sep 10 13:17:18.334419 ip-10-0-139-174 kubenswrapper[2557]: W0910 13:17:18.333546 2557 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Sep 10 13:17:18.334419 ip-10-0-139-174 kubenswrapper[2557]: W0910 13:17:18.333549 2557 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Sep 10 13:17:18.334904 ip-10-0-139-174 kubenswrapper[2557]: W0910 13:17:18.333552 2557 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Sep 10 13:17:18.334904 ip-10-0-139-174 kubenswrapper[2557]: W0910 13:17:18.333555 2557 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Sep 10 13:17:18.334904 ip-10-0-139-174 kubenswrapper[2557]: W0910 13:17:18.333557 2557 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Sep 10 13:17:18.334904 ip-10-0-139-174 kubenswrapper[2557]: W0910 13:17:18.333560 2557 feature_gate.go:328] unrecognized feature gate: InsightsConfig Sep 10 13:17:18.334904 ip-10-0-139-174 kubenswrapper[2557]: W0910 13:17:18.333562 2557 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Sep 10 13:17:18.334904 ip-10-0-139-174 kubenswrapper[2557]: W0910 13:17:18.333564 2557 feature_gate.go:328] unrecognized feature gate: OVNObservability Sep 10 13:17:18.334904 ip-10-0-139-174 kubenswrapper[2557]: W0910 13:17:18.333566 2557 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Sep 10 13:17:18.334904 ip-10-0-139-174 kubenswrapper[2557]: W0910 13:17:18.333569 2557 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Sep 10 13:17:18.334904 ip-10-0-139-174 kubenswrapper[2557]: W0910 13:17:18.333571 2557 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Sep 10 13:17:18.334904 ip-10-0-139-174 kubenswrapper[2557]: W0910 13:17:18.333574 2557 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Sep 10 13:17:18.334904 ip-10-0-139-174 kubenswrapper[2557]: W0910 13:17:18.333576 2557 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Sep 10 13:17:18.334904 ip-10-0-139-174 kubenswrapper[2557]: W0910 13:17:18.333579 2557 feature_gate.go:328] unrecognized feature gate: EventTTL Sep 10 13:17:18.334904 ip-10-0-139-174 kubenswrapper[2557]: W0910 13:17:18.333582 2557 feature_gate.go:328] unrecognized feature gate: PinnedImages Sep 10 13:17:18.334904 ip-10-0-139-174 kubenswrapper[2557]: W0910 13:17:18.333584 2557 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Sep 10 13:17:18.334904 ip-10-0-139-174 kubenswrapper[2557]: W0910 13:17:18.333587 2557 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Sep 10 13:17:18.334904 ip-10-0-139-174 kubenswrapper[2557]: W0910 13:17:18.333589 2557 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Sep 10 13:17:18.334904 ip-10-0-139-174 kubenswrapper[2557]: W0910 13:17:18.333591 2557 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Sep 10 13:17:18.334904 ip-10-0-139-174 kubenswrapper[2557]: W0910 13:17:18.333594 2557 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Sep 10 13:17:18.334904 ip-10-0-139-174 kubenswrapper[2557]: W0910 13:17:18.333596 2557 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Sep 10 13:17:18.334904 ip-10-0-139-174 kubenswrapper[2557]: W0910 13:17:18.333599 2557 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Sep 10 13:17:18.335409 ip-10-0-139-174 kubenswrapper[2557]: W0910 13:17:18.333601 2557 feature_gate.go:328] unrecognized feature gate: SignatureStores Sep 10 13:17:18.335409 ip-10-0-139-174 kubenswrapper[2557]: W0910 13:17:18.333603 2557 feature_gate.go:328] unrecognized feature gate: Example2 Sep 10 13:17:18.335409 ip-10-0-139-174 kubenswrapper[2557]: W0910 13:17:18.333606 2557 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Sep 10 13:17:18.335409 ip-10-0-139-174 kubenswrapper[2557]: W0910 13:17:18.333608 2557 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Sep 10 13:17:18.335409 ip-10-0-139-174 kubenswrapper[2557]: W0910 13:17:18.333611 2557 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Sep 10 13:17:18.335409 ip-10-0-139-174 kubenswrapper[2557]: W0910 13:17:18.333613 2557 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Sep 10 13:17:18.335409 ip-10-0-139-174 kubenswrapper[2557]: W0910 13:17:18.333615 2557 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Sep 10 13:17:18.335409 ip-10-0-139-174 kubenswrapper[2557]: W0910 13:17:18.333619 2557 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Sep 10 13:17:18.335409 ip-10-0-139-174 kubenswrapper[2557]: W0910 13:17:18.333621 2557 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Sep 10 13:17:18.335409 ip-10-0-139-174 kubenswrapper[2557]: W0910 13:17:18.333624 2557 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Sep 10 13:17:18.335409 ip-10-0-139-174 kubenswrapper[2557]: W0910 13:17:18.333626 2557 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Sep 10 13:17:18.335409 ip-10-0-139-174 kubenswrapper[2557]: W0910 13:17:18.333629 2557 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Sep 10 13:17:18.335409 ip-10-0-139-174 kubenswrapper[2557]: W0910 13:17:18.333631 2557 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Sep 10 13:17:18.335409 ip-10-0-139-174 kubenswrapper[2557]: W0910 13:17:18.333634 2557 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Sep 10 13:17:18.335409 ip-10-0-139-174 kubenswrapper[2557]: W0910 13:17:18.333636 2557 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Sep 10 13:17:18.335409 ip-10-0-139-174 kubenswrapper[2557]: W0910 13:17:18.333639 2557 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Sep 10 13:17:18.335409 ip-10-0-139-174 kubenswrapper[2557]: W0910 13:17:18.333641 2557 feature_gate.go:328] unrecognized feature gate: DualReplica Sep 10 13:17:18.335409 ip-10-0-139-174 kubenswrapper[2557]: W0910 13:17:18.333644 2557 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Sep 10 13:17:18.335409 ip-10-0-139-174 kubenswrapper[2557]: W0910 13:17:18.333646 2557 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Sep 10 13:17:18.335857 ip-10-0-139-174 kubenswrapper[2557]: W0910 13:17:18.334313 2557 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Sep 10 13:17:18.335857 ip-10-0-139-174 kubenswrapper[2557]: W0910 13:17:18.334320 2557 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Sep 10 13:17:18.335857 ip-10-0-139-174 kubenswrapper[2557]: W0910 13:17:18.334323 2557 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Sep 10 13:17:18.335857 ip-10-0-139-174 kubenswrapper[2557]: W0910 13:17:18.334325 2557 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Sep 10 13:17:18.335857 ip-10-0-139-174 kubenswrapper[2557]: W0910 13:17:18.334328 2557 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Sep 10 13:17:18.335857 ip-10-0-139-174 kubenswrapper[2557]: W0910 13:17:18.334331 2557 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Sep 10 13:17:18.335857 ip-10-0-139-174 kubenswrapper[2557]: W0910 13:17:18.334334 2557 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Sep 10 13:17:18.335857 ip-10-0-139-174 kubenswrapper[2557]: W0910 13:17:18.334337 2557 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Sep 10 13:17:18.335857 ip-10-0-139-174 kubenswrapper[2557]: W0910 13:17:18.334339 2557 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Sep 10 13:17:18.335857 ip-10-0-139-174 kubenswrapper[2557]: W0910 13:17:18.334342 2557 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Sep 10 13:17:18.335857 ip-10-0-139-174 kubenswrapper[2557]: W0910 13:17:18.334344 2557 feature_gate.go:328] unrecognized feature gate: InsightsConfig Sep 10 13:17:18.335857 ip-10-0-139-174 kubenswrapper[2557]: W0910 13:17:18.334349 2557 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Sep 10 13:17:18.335857 ip-10-0-139-174 kubenswrapper[2557]: W0910 13:17:18.334352 2557 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Sep 10 13:17:18.335857 ip-10-0-139-174 kubenswrapper[2557]: W0910 13:17:18.334366 2557 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Sep 10 13:17:18.335857 ip-10-0-139-174 kubenswrapper[2557]: W0910 13:17:18.334370 2557 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Sep 10 13:17:18.335857 ip-10-0-139-174 kubenswrapper[2557]: W0910 13:17:18.334374 2557 feature_gate.go:328] unrecognized feature gate: DualReplica Sep 10 13:17:18.335857 ip-10-0-139-174 kubenswrapper[2557]: W0910 13:17:18.334377 2557 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Sep 10 13:17:18.335857 ip-10-0-139-174 kubenswrapper[2557]: W0910 13:17:18.334379 2557 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Sep 10 13:17:18.335857 ip-10-0-139-174 kubenswrapper[2557]: W0910 13:17:18.334382 2557 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Sep 10 13:17:18.336329 ip-10-0-139-174 kubenswrapper[2557]: W0910 13:17:18.334384 2557 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Sep 10 13:17:18.336329 ip-10-0-139-174 kubenswrapper[2557]: W0910 13:17:18.334388 2557 feature_gate.go:328] unrecognized feature gate: Example2 Sep 10 13:17:18.336329 ip-10-0-139-174 kubenswrapper[2557]: W0910 13:17:18.334391 2557 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Sep 10 13:17:18.336329 ip-10-0-139-174 kubenswrapper[2557]: W0910 13:17:18.334393 2557 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Sep 10 13:17:18.336329 ip-10-0-139-174 kubenswrapper[2557]: W0910 13:17:18.334396 2557 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Sep 10 13:17:18.336329 ip-10-0-139-174 kubenswrapper[2557]: W0910 13:17:18.334399 2557 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Sep 10 13:17:18.336329 ip-10-0-139-174 kubenswrapper[2557]: W0910 13:17:18.334401 2557 feature_gate.go:328] unrecognized feature gate: KMSEncryption Sep 10 13:17:18.336329 ip-10-0-139-174 kubenswrapper[2557]: W0910 13:17:18.334404 2557 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Sep 10 13:17:18.336329 ip-10-0-139-174 kubenswrapper[2557]: W0910 13:17:18.334406 2557 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Sep 10 13:17:18.336329 ip-10-0-139-174 kubenswrapper[2557]: W0910 13:17:18.334409 2557 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Sep 10 13:17:18.336329 ip-10-0-139-174 kubenswrapper[2557]: W0910 13:17:18.334411 2557 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Sep 10 13:17:18.336329 ip-10-0-139-174 kubenswrapper[2557]: W0910 13:17:18.334414 2557 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Sep 10 13:17:18.336329 ip-10-0-139-174 kubenswrapper[2557]: W0910 13:17:18.334416 2557 feature_gate.go:328] unrecognized feature gate: SignatureStores Sep 10 13:17:18.336329 ip-10-0-139-174 kubenswrapper[2557]: W0910 13:17:18.334419 2557 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Sep 10 13:17:18.336329 ip-10-0-139-174 kubenswrapper[2557]: W0910 13:17:18.334421 2557 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Sep 10 13:17:18.336329 ip-10-0-139-174 kubenswrapper[2557]: W0910 13:17:18.334424 2557 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Sep 10 13:17:18.336329 ip-10-0-139-174 kubenswrapper[2557]: W0910 13:17:18.334426 2557 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Sep 10 13:17:18.336329 ip-10-0-139-174 kubenswrapper[2557]: W0910 13:17:18.334429 2557 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Sep 10 13:17:18.336329 ip-10-0-139-174 kubenswrapper[2557]: W0910 13:17:18.334431 2557 feature_gate.go:328] unrecognized feature gate: GatewayAPI Sep 10 13:17:18.336329 ip-10-0-139-174 kubenswrapper[2557]: W0910 13:17:18.334435 2557 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Sep 10 13:17:18.336807 ip-10-0-139-174 kubenswrapper[2557]: W0910 13:17:18.334437 2557 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Sep 10 13:17:18.336807 ip-10-0-139-174 kubenswrapper[2557]: W0910 13:17:18.334440 2557 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Sep 10 13:17:18.336807 ip-10-0-139-174 kubenswrapper[2557]: W0910 13:17:18.334442 2557 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Sep 10 13:17:18.336807 ip-10-0-139-174 kubenswrapper[2557]: W0910 13:17:18.334445 2557 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Sep 10 13:17:18.336807 ip-10-0-139-174 kubenswrapper[2557]: W0910 13:17:18.334447 2557 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Sep 10 13:17:18.336807 ip-10-0-139-174 kubenswrapper[2557]: W0910 13:17:18.334450 2557 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Sep 10 13:17:18.336807 ip-10-0-139-174 kubenswrapper[2557]: W0910 13:17:18.334452 2557 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Sep 10 13:17:18.336807 ip-10-0-139-174 kubenswrapper[2557]: W0910 13:17:18.334454 2557 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Sep 10 13:17:18.336807 ip-10-0-139-174 kubenswrapper[2557]: W0910 13:17:18.334457 2557 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Sep 10 13:17:18.336807 ip-10-0-139-174 kubenswrapper[2557]: W0910 13:17:18.334459 2557 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Sep 10 13:17:18.336807 ip-10-0-139-174 kubenswrapper[2557]: W0910 13:17:18.334461 2557 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Sep 10 13:17:18.336807 ip-10-0-139-174 kubenswrapper[2557]: W0910 13:17:18.334464 2557 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Sep 10 13:17:18.336807 ip-10-0-139-174 kubenswrapper[2557]: W0910 13:17:18.334467 2557 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Sep 10 13:17:18.336807 ip-10-0-139-174 kubenswrapper[2557]: W0910 13:17:18.334473 2557 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Sep 10 13:17:18.336807 ip-10-0-139-174 kubenswrapper[2557]: W0910 13:17:18.334477 2557 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Sep 10 13:17:18.336807 ip-10-0-139-174 kubenswrapper[2557]: W0910 13:17:18.334479 2557 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Sep 10 13:17:18.336807 ip-10-0-139-174 kubenswrapper[2557]: W0910 13:17:18.334482 2557 feature_gate.go:328] unrecognized feature gate: PinnedImages Sep 10 13:17:18.336807 ip-10-0-139-174 kubenswrapper[2557]: W0910 13:17:18.334485 2557 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Sep 10 13:17:18.336807 ip-10-0-139-174 kubenswrapper[2557]: W0910 13:17:18.334487 2557 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Sep 10 13:17:18.337292 ip-10-0-139-174 kubenswrapper[2557]: W0910 13:17:18.334489 2557 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Sep 10 13:17:18.337292 ip-10-0-139-174 kubenswrapper[2557]: W0910 13:17:18.334492 2557 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Sep 10 13:17:18.337292 ip-10-0-139-174 kubenswrapper[2557]: W0910 13:17:18.334494 2557 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Sep 10 13:17:18.337292 ip-10-0-139-174 kubenswrapper[2557]: W0910 13:17:18.334497 2557 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Sep 10 13:17:18.337292 ip-10-0-139-174 kubenswrapper[2557]: W0910 13:17:18.334499 2557 feature_gate.go:328] unrecognized feature gate: OSStreams Sep 10 13:17:18.337292 ip-10-0-139-174 kubenswrapper[2557]: W0910 13:17:18.334502 2557 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Sep 10 13:17:18.337292 ip-10-0-139-174 kubenswrapper[2557]: W0910 13:17:18.334504 2557 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Sep 10 13:17:18.337292 ip-10-0-139-174 kubenswrapper[2557]: W0910 13:17:18.334507 2557 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Sep 10 13:17:18.337292 ip-10-0-139-174 kubenswrapper[2557]: W0910 13:17:18.334509 2557 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Sep 10 13:17:18.337292 ip-10-0-139-174 kubenswrapper[2557]: W0910 13:17:18.334512 2557 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Sep 10 13:17:18.337292 ip-10-0-139-174 kubenswrapper[2557]: W0910 13:17:18.334514 2557 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Sep 10 13:17:18.337292 ip-10-0-139-174 kubenswrapper[2557]: W0910 13:17:18.334516 2557 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Sep 10 13:17:18.337292 ip-10-0-139-174 kubenswrapper[2557]: W0910 13:17:18.334519 2557 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Sep 10 13:17:18.337292 ip-10-0-139-174 kubenswrapper[2557]: W0910 13:17:18.334521 2557 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Sep 10 13:17:18.337292 ip-10-0-139-174 kubenswrapper[2557]: W0910 13:17:18.334524 2557 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Sep 10 13:17:18.337292 ip-10-0-139-174 kubenswrapper[2557]: W0910 13:17:18.334526 2557 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Sep 10 13:17:18.337292 ip-10-0-139-174 kubenswrapper[2557]: W0910 13:17:18.334529 2557 feature_gate.go:328] unrecognized feature gate: Example Sep 10 13:17:18.337292 ip-10-0-139-174 kubenswrapper[2557]: W0910 13:17:18.334532 2557 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Sep 10 13:17:18.337292 ip-10-0-139-174 kubenswrapper[2557]: W0910 13:17:18.334535 2557 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Sep 10 13:17:18.337292 ip-10-0-139-174 kubenswrapper[2557]: W0910 13:17:18.334537 2557 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Sep 10 13:17:18.337882 ip-10-0-139-174 kubenswrapper[2557]: W0910 13:17:18.334540 2557 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Sep 10 13:17:18.337882 ip-10-0-139-174 kubenswrapper[2557]: W0910 13:17:18.334542 2557 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Sep 10 13:17:18.337882 ip-10-0-139-174 kubenswrapper[2557]: W0910 13:17:18.334544 2557 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Sep 10 13:17:18.337882 ip-10-0-139-174 kubenswrapper[2557]: W0910 13:17:18.334547 2557 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Sep 10 13:17:18.337882 ip-10-0-139-174 kubenswrapper[2557]: W0910 13:17:18.334549 2557 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Sep 10 13:17:18.337882 ip-10-0-139-174 kubenswrapper[2557]: W0910 13:17:18.334551 2557 feature_gate.go:328] unrecognized feature gate: OVNObservability Sep 10 13:17:18.337882 ip-10-0-139-174 kubenswrapper[2557]: W0910 13:17:18.334555 2557 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Sep 10 13:17:18.337882 ip-10-0-139-174 kubenswrapper[2557]: W0910 13:17:18.334558 2557 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Sep 10 13:17:18.337882 ip-10-0-139-174 kubenswrapper[2557]: W0910 13:17:18.334560 2557 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Sep 10 13:17:18.337882 ip-10-0-139-174 kubenswrapper[2557]: W0910 13:17:18.334562 2557 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Sep 10 13:17:18.337882 ip-10-0-139-174 kubenswrapper[2557]: W0910 13:17:18.334565 2557 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Sep 10 13:17:18.337882 ip-10-0-139-174 kubenswrapper[2557]: W0910 13:17:18.334567 2557 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Sep 10 13:17:18.337882 ip-10-0-139-174 kubenswrapper[2557]: W0910 13:17:18.334570 2557 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Sep 10 13:17:18.337882 ip-10-0-139-174 kubenswrapper[2557]: W0910 13:17:18.334572 2557 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Sep 10 13:17:18.337882 ip-10-0-139-174 kubenswrapper[2557]: W0910 13:17:18.334575 2557 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Sep 10 13:17:18.337882 ip-10-0-139-174 kubenswrapper[2557]: W0910 13:17:18.334577 2557 feature_gate.go:328] unrecognized feature gate: EventTTL Sep 10 13:17:18.337882 ip-10-0-139-174 kubenswrapper[2557]: W0910 13:17:18.334580 2557 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Sep 10 13:17:18.337882 ip-10-0-139-174 kubenswrapper[2557]: W0910 13:17:18.334582 2557 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Sep 10 13:17:18.337882 ip-10-0-139-174 kubenswrapper[2557]: W0910 13:17:18.334585 2557 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Sep 10 13:17:18.337882 ip-10-0-139-174 kubenswrapper[2557]: W0910 13:17:18.334587 2557 feature_gate.go:328] unrecognized feature gate: NewOLM Sep 10 13:17:18.338398 ip-10-0-139-174 kubenswrapper[2557]: W0910 13:17:18.334590 2557 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Sep 10 13:17:18.338398 ip-10-0-139-174 kubenswrapper[2557]: W0910 13:17:18.334923 2557 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Sep 10 13:17:18.338398 ip-10-0-139-174 kubenswrapper[2557]: W0910 13:17:18.334928 2557 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Sep 10 13:17:18.338398 ip-10-0-139-174 kubenswrapper[2557]: W0910 13:17:18.334931 2557 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Sep 10 13:17:18.338398 ip-10-0-139-174 kubenswrapper[2557]: W0910 13:17:18.334936 2557 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Sep 10 13:17:18.338398 ip-10-0-139-174 kubenswrapper[2557]: W0910 13:17:18.334940 2557 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Sep 10 13:17:18.338398 ip-10-0-139-174 kubenswrapper[2557]: W0910 13:17:18.334944 2557 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Sep 10 13:17:18.338398 ip-10-0-139-174 kubenswrapper[2557]: W0910 13:17:18.334947 2557 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Sep 10 13:17:18.338398 ip-10-0-139-174 kubenswrapper[2557]: W0910 13:17:18.334950 2557 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Sep 10 13:17:18.338398 ip-10-0-139-174 kubenswrapper[2557]: W0910 13:17:18.334952 2557 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Sep 10 13:17:18.338398 ip-10-0-139-174 kubenswrapper[2557]: W0910 13:17:18.334955 2557 feature_gate.go:328] unrecognized feature gate: InsightsConfig Sep 10 13:17:18.338398 ip-10-0-139-174 kubenswrapper[2557]: W0910 13:17:18.334958 2557 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Sep 10 13:17:18.338398 ip-10-0-139-174 kubenswrapper[2557]: W0910 13:17:18.334961 2557 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Sep 10 13:17:18.338398 ip-10-0-139-174 kubenswrapper[2557]: W0910 13:17:18.334965 2557 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Sep 10 13:17:18.338398 ip-10-0-139-174 kubenswrapper[2557]: W0910 13:17:18.334969 2557 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Sep 10 13:17:18.338398 ip-10-0-139-174 kubenswrapper[2557]: W0910 13:17:18.334972 2557 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Sep 10 13:17:18.338398 ip-10-0-139-174 kubenswrapper[2557]: W0910 13:17:18.334974 2557 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Sep 10 13:17:18.338398 ip-10-0-139-174 kubenswrapper[2557]: W0910 13:17:18.334977 2557 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Sep 10 13:17:18.338398 ip-10-0-139-174 kubenswrapper[2557]: W0910 13:17:18.334982 2557 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Sep 10 13:17:18.338863 ip-10-0-139-174 kubenswrapper[2557]: W0910 13:17:18.334985 2557 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Sep 10 13:17:18.338863 ip-10-0-139-174 kubenswrapper[2557]: W0910 13:17:18.334988 2557 feature_gate.go:328] unrecognized feature gate: NewOLM Sep 10 13:17:18.338863 ip-10-0-139-174 kubenswrapper[2557]: W0910 13:17:18.334990 2557 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Sep 10 13:17:18.338863 ip-10-0-139-174 kubenswrapper[2557]: W0910 13:17:18.334993 2557 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Sep 10 13:17:18.338863 ip-10-0-139-174 kubenswrapper[2557]: W0910 13:17:18.334997 2557 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Sep 10 13:17:18.338863 ip-10-0-139-174 kubenswrapper[2557]: W0910 13:17:18.335000 2557 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Sep 10 13:17:18.338863 ip-10-0-139-174 kubenswrapper[2557]: W0910 13:17:18.335003 2557 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Sep 10 13:17:18.338863 ip-10-0-139-174 kubenswrapper[2557]: W0910 13:17:18.335006 2557 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Sep 10 13:17:18.338863 ip-10-0-139-174 kubenswrapper[2557]: W0910 13:17:18.335009 2557 feature_gate.go:328] unrecognized feature gate: OSStreams Sep 10 13:17:18.338863 ip-10-0-139-174 kubenswrapper[2557]: W0910 13:17:18.335011 2557 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Sep 10 13:17:18.338863 ip-10-0-139-174 kubenswrapper[2557]: W0910 13:17:18.335014 2557 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Sep 10 13:17:18.338863 ip-10-0-139-174 kubenswrapper[2557]: W0910 13:17:18.335017 2557 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Sep 10 13:17:18.338863 ip-10-0-139-174 kubenswrapper[2557]: W0910 13:17:18.335020 2557 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Sep 10 13:17:18.338863 ip-10-0-139-174 kubenswrapper[2557]: W0910 13:17:18.335022 2557 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Sep 10 13:17:18.338863 ip-10-0-139-174 kubenswrapper[2557]: W0910 13:17:18.335025 2557 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Sep 10 13:17:18.338863 ip-10-0-139-174 kubenswrapper[2557]: W0910 13:17:18.335028 2557 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Sep 10 13:17:18.338863 ip-10-0-139-174 kubenswrapper[2557]: W0910 13:17:18.335030 2557 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Sep 10 13:17:18.338863 ip-10-0-139-174 kubenswrapper[2557]: W0910 13:17:18.335033 2557 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Sep 10 13:17:18.338863 ip-10-0-139-174 kubenswrapper[2557]: W0910 13:17:18.335035 2557 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Sep 10 13:17:18.339373 ip-10-0-139-174 kubenswrapper[2557]: W0910 13:17:18.335039 2557 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Sep 10 13:17:18.339373 ip-10-0-139-174 kubenswrapper[2557]: W0910 13:17:18.335041 2557 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Sep 10 13:17:18.339373 ip-10-0-139-174 kubenswrapper[2557]: W0910 13:17:18.335044 2557 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Sep 10 13:17:18.339373 ip-10-0-139-174 kubenswrapper[2557]: W0910 13:17:18.335047 2557 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Sep 10 13:17:18.339373 ip-10-0-139-174 kubenswrapper[2557]: W0910 13:17:18.335049 2557 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Sep 10 13:17:18.339373 ip-10-0-139-174 kubenswrapper[2557]: W0910 13:17:18.335052 2557 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Sep 10 13:17:18.339373 ip-10-0-139-174 kubenswrapper[2557]: W0910 13:17:18.335055 2557 feature_gate.go:328] unrecognized feature gate: Example Sep 10 13:17:18.339373 ip-10-0-139-174 kubenswrapper[2557]: W0910 13:17:18.335057 2557 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Sep 10 13:17:18.339373 ip-10-0-139-174 kubenswrapper[2557]: W0910 13:17:18.335060 2557 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Sep 10 13:17:18.339373 ip-10-0-139-174 kubenswrapper[2557]: W0910 13:17:18.335077 2557 feature_gate.go:328] unrecognized feature gate: PinnedImages Sep 10 13:17:18.339373 ip-10-0-139-174 kubenswrapper[2557]: W0910 13:17:18.335080 2557 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Sep 10 13:17:18.339373 ip-10-0-139-174 kubenswrapper[2557]: W0910 13:17:18.335082 2557 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Sep 10 13:17:18.339373 ip-10-0-139-174 kubenswrapper[2557]: W0910 13:17:18.335086 2557 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Sep 10 13:17:18.339373 ip-10-0-139-174 kubenswrapper[2557]: W0910 13:17:18.335088 2557 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Sep 10 13:17:18.339373 ip-10-0-139-174 kubenswrapper[2557]: W0910 13:17:18.335091 2557 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Sep 10 13:17:18.339373 ip-10-0-139-174 kubenswrapper[2557]: W0910 13:17:18.335093 2557 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Sep 10 13:17:18.339373 ip-10-0-139-174 kubenswrapper[2557]: W0910 13:17:18.335096 2557 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Sep 10 13:17:18.339373 ip-10-0-139-174 kubenswrapper[2557]: W0910 13:17:18.335098 2557 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Sep 10 13:17:18.339373 ip-10-0-139-174 kubenswrapper[2557]: W0910 13:17:18.335101 2557 feature_gate.go:328] unrecognized feature gate: GatewayAPI Sep 10 13:17:18.339373 ip-10-0-139-174 kubenswrapper[2557]: W0910 13:17:18.335103 2557 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Sep 10 13:17:18.339846 ip-10-0-139-174 kubenswrapper[2557]: W0910 13:17:18.335106 2557 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Sep 10 13:17:18.339846 ip-10-0-139-174 kubenswrapper[2557]: W0910 13:17:18.335108 2557 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Sep 10 13:17:18.339846 ip-10-0-139-174 kubenswrapper[2557]: W0910 13:17:18.335111 2557 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Sep 10 13:17:18.339846 ip-10-0-139-174 kubenswrapper[2557]: W0910 13:17:18.335113 2557 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Sep 10 13:17:18.339846 ip-10-0-139-174 kubenswrapper[2557]: W0910 13:17:18.335117 2557 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Sep 10 13:17:18.339846 ip-10-0-139-174 kubenswrapper[2557]: W0910 13:17:18.335119 2557 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Sep 10 13:17:18.339846 ip-10-0-139-174 kubenswrapper[2557]: W0910 13:17:18.335121 2557 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Sep 10 13:17:18.339846 ip-10-0-139-174 kubenswrapper[2557]: W0910 13:17:18.335124 2557 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Sep 10 13:17:18.339846 ip-10-0-139-174 kubenswrapper[2557]: W0910 13:17:18.335126 2557 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Sep 10 13:17:18.339846 ip-10-0-139-174 kubenswrapper[2557]: W0910 13:17:18.335129 2557 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Sep 10 13:17:18.339846 ip-10-0-139-174 kubenswrapper[2557]: W0910 13:17:18.335131 2557 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Sep 10 13:17:18.339846 ip-10-0-139-174 kubenswrapper[2557]: W0910 13:17:18.335133 2557 feature_gate.go:328] unrecognized feature gate: OVNObservability Sep 10 13:17:18.339846 ip-10-0-139-174 kubenswrapper[2557]: W0910 13:17:18.335136 2557 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Sep 10 13:17:18.339846 ip-10-0-139-174 kubenswrapper[2557]: W0910 13:17:18.335138 2557 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Sep 10 13:17:18.339846 ip-10-0-139-174 kubenswrapper[2557]: W0910 13:17:18.335141 2557 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Sep 10 13:17:18.339846 ip-10-0-139-174 kubenswrapper[2557]: W0910 13:17:18.335144 2557 feature_gate.go:328] unrecognized feature gate: DualReplica Sep 10 13:17:18.339846 ip-10-0-139-174 kubenswrapper[2557]: W0910 13:17:18.335146 2557 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Sep 10 13:17:18.339846 ip-10-0-139-174 kubenswrapper[2557]: W0910 13:17:18.335149 2557 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Sep 10 13:17:18.339846 ip-10-0-139-174 kubenswrapper[2557]: W0910 13:17:18.335151 2557 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Sep 10 13:17:18.340313 ip-10-0-139-174 kubenswrapper[2557]: W0910 13:17:18.335154 2557 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Sep 10 13:17:18.340313 ip-10-0-139-174 kubenswrapper[2557]: W0910 13:17:18.335156 2557 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Sep 10 13:17:18.340313 ip-10-0-139-174 kubenswrapper[2557]: W0910 13:17:18.335159 2557 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Sep 10 13:17:18.340313 ip-10-0-139-174 kubenswrapper[2557]: W0910 13:17:18.335161 2557 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Sep 10 13:17:18.340313 ip-10-0-139-174 kubenswrapper[2557]: W0910 13:17:18.335164 2557 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Sep 10 13:17:18.340313 ip-10-0-139-174 kubenswrapper[2557]: W0910 13:17:18.335168 2557 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Sep 10 13:17:18.340313 ip-10-0-139-174 kubenswrapper[2557]: W0910 13:17:18.335170 2557 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Sep 10 13:17:18.340313 ip-10-0-139-174 kubenswrapper[2557]: W0910 13:17:18.335173 2557 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Sep 10 13:17:18.340313 ip-10-0-139-174 kubenswrapper[2557]: W0910 13:17:18.335175 2557 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Sep 10 13:17:18.340313 ip-10-0-139-174 kubenswrapper[2557]: W0910 13:17:18.335178 2557 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Sep 10 13:17:18.340313 ip-10-0-139-174 kubenswrapper[2557]: W0910 13:17:18.335180 2557 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Sep 10 13:17:18.340313 ip-10-0-139-174 kubenswrapper[2557]: W0910 13:17:18.335182 2557 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Sep 10 13:17:18.340313 ip-10-0-139-174 kubenswrapper[2557]: W0910 13:17:18.335185 2557 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Sep 10 13:17:18.340313 ip-10-0-139-174 kubenswrapper[2557]: W0910 13:17:18.335187 2557 feature_gate.go:328] unrecognized feature gate: Example2 Sep 10 13:17:18.340313 ip-10-0-139-174 kubenswrapper[2557]: W0910 13:17:18.335190 2557 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Sep 10 13:17:18.340313 ip-10-0-139-174 kubenswrapper[2557]: W0910 13:17:18.335192 2557 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Sep 10 13:17:18.340313 ip-10-0-139-174 kubenswrapper[2557]: W0910 13:17:18.335194 2557 feature_gate.go:328] unrecognized feature gate: EventTTL Sep 10 13:17:18.340313 ip-10-0-139-174 kubenswrapper[2557]: W0910 13:17:18.335197 2557 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Sep 10 13:17:18.340313 ip-10-0-139-174 kubenswrapper[2557]: W0910 13:17:18.335199 2557 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Sep 10 13:17:18.340313 ip-10-0-139-174 kubenswrapper[2557]: W0910 13:17:18.335202 2557 feature_gate.go:328] unrecognized feature gate: KMSEncryption Sep 10 13:17:18.340830 ip-10-0-139-174 kubenswrapper[2557]: W0910 13:17:18.335204 2557 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Sep 10 13:17:18.340830 ip-10-0-139-174 kubenswrapper[2557]: W0910 13:17:18.335207 2557 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Sep 10 13:17:18.340830 ip-10-0-139-174 kubenswrapper[2557]: W0910 13:17:18.335209 2557 feature_gate.go:328] unrecognized feature gate: SignatureStores Sep 10 13:17:18.340830 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:17:18.335280 2557 flags.go:64] FLAG: --address="0.0.0.0" Sep 10 13:17:18.340830 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:17:18.335318 2557 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]" Sep 10 13:17:18.340830 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:17:18.335324 2557 flags.go:64] FLAG: --anonymous-auth="true" Sep 10 13:17:18.340830 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:17:18.335328 2557 flags.go:64] FLAG: --application-metrics-count-limit="100" Sep 10 13:17:18.340830 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:17:18.335333 2557 flags.go:64] FLAG: --authentication-token-webhook="false" Sep 10 13:17:18.340830 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:17:18.335336 2557 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s" Sep 10 13:17:18.340830 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:17:18.335340 2557 flags.go:64] FLAG: --authorization-mode="AlwaysAllow" Sep 10 13:17:18.340830 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:17:18.335344 2557 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s" Sep 10 13:17:18.340830 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:17:18.335347 2557 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s" Sep 10 13:17:18.340830 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:17:18.335350 2557 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id" Sep 10 13:17:18.340830 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:17:18.335353 2557 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig" Sep 10 13:17:18.340830 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:17:18.335357 2557 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki" Sep 10 13:17:18.340830 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:17:18.335360 2557 flags.go:64] FLAG: --cgroup-driver="cgroupfs" Sep 10 13:17:18.340830 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:17:18.335363 2557 flags.go:64] FLAG: --cgroup-root="" Sep 10 13:17:18.340830 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:17:18.335367 2557 flags.go:64] FLAG: --cgroups-per-qos="true" Sep 10 13:17:18.340830 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:17:18.335370 2557 flags.go:64] FLAG: --client-ca-file="" Sep 10 13:17:18.340830 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:17:18.335373 2557 flags.go:64] FLAG: --cloud-provider="external" Sep 10 13:17:18.340830 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:17:18.335376 2557 flags.go:64] FLAG: --cluster-dns="[]" Sep 10 13:17:18.340830 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:17:18.335379 2557 flags.go:64] FLAG: --cluster-domain="" Sep 10 13:17:18.340830 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:17:18.335382 2557 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf" Sep 10 13:17:18.341390 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:17:18.335384 2557 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d" Sep 10 13:17:18.341390 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:17:18.335387 2557 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json" Sep 10 13:17:18.341390 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:17:18.335390 2557 flags.go:64] FLAG: --container-log-max-files="5" Sep 10 13:17:18.341390 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:17:18.335394 2557 flags.go:64] FLAG: --container-log-max-size="10Mi" Sep 10 13:17:18.341390 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:17:18.335397 2557 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock" Sep 10 13:17:18.341390 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:17:18.335400 2557 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock" Sep 10 13:17:18.341390 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:17:18.335403 2557 flags.go:64] FLAG: --containerd-namespace="k8s.io" Sep 10 13:17:18.341390 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:17:18.335406 2557 flags.go:64] FLAG: --contention-profiling="false" Sep 10 13:17:18.341390 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:17:18.335409 2557 flags.go:64] FLAG: --cpu-cfs-quota="true" Sep 10 13:17:18.341390 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:17:18.335412 2557 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms" Sep 10 13:17:18.341390 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:17:18.335415 2557 flags.go:64] FLAG: --cpu-manager-policy="none" Sep 10 13:17:18.341390 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:17:18.335418 2557 flags.go:64] FLAG: --cpu-manager-policy-options="" Sep 10 13:17:18.341390 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:17:18.335421 2557 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s" Sep 10 13:17:18.341390 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:17:18.335424 2557 flags.go:64] FLAG: --enable-controller-attach-detach="true" Sep 10 13:17:18.341390 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:17:18.335427 2557 flags.go:64] FLAG: --enable-debugging-handlers="true" Sep 10 13:17:18.341390 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:17:18.335430 2557 flags.go:64] FLAG: --enable-load-reader="false" Sep 10 13:17:18.341390 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:17:18.335432 2557 flags.go:64] FLAG: --enable-server="true" Sep 10 13:17:18.341390 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:17:18.335438 2557 flags.go:64] FLAG: --enforce-node-allocatable="[pods]" Sep 10 13:17:18.341390 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:17:18.335443 2557 flags.go:64] FLAG: --event-burst="100" Sep 10 13:17:18.341390 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:17:18.335446 2557 flags.go:64] FLAG: --event-qps="50" Sep 10 13:17:18.341390 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:17:18.335449 2557 flags.go:64] FLAG: --event-storage-age-limit="default=0" Sep 10 13:17:18.341390 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:17:18.335452 2557 flags.go:64] FLAG: --event-storage-event-limit="default=0" Sep 10 13:17:18.341390 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:17:18.335456 2557 flags.go:64] FLAG: --eviction-hard="" Sep 10 13:17:18.341390 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:17:18.335460 2557 flags.go:64] FLAG: --eviction-max-pod-grace-period="0" Sep 10 13:17:18.341945 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:17:18.335462 2557 flags.go:64] FLAG: --eviction-minimum-reclaim="" Sep 10 13:17:18.341945 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:17:18.335465 2557 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s" Sep 10 13:17:18.341945 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:17:18.335469 2557 flags.go:64] FLAG: --eviction-soft="" Sep 10 13:17:18.341945 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:17:18.335472 2557 flags.go:64] FLAG: --eviction-soft-grace-period="" Sep 10 13:17:18.341945 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:17:18.335475 2557 flags.go:64] FLAG: --exit-on-lock-contention="false" Sep 10 13:17:18.341945 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:17:18.335477 2557 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false" Sep 10 13:17:18.341945 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:17:18.335480 2557 flags.go:64] FLAG: --experimental-mounter-path="" Sep 10 13:17:18.341945 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:17:18.335483 2557 flags.go:64] FLAG: --fail-cgroupv1="false" Sep 10 13:17:18.341945 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:17:18.335485 2557 flags.go:64] FLAG: --fail-swap-on="true" Sep 10 13:17:18.341945 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:17:18.335488 2557 flags.go:64] FLAG: --feature-gates="" Sep 10 13:17:18.341945 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:17:18.335492 2557 flags.go:64] FLAG: --file-check-frequency="20s" Sep 10 13:17:18.341945 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:17:18.335494 2557 flags.go:64] FLAG: --global-housekeeping-interval="1m0s" Sep 10 13:17:18.341945 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:17:18.335497 2557 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge" Sep 10 13:17:18.341945 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:17:18.335500 2557 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1" Sep 10 13:17:18.341945 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:17:18.335503 2557 flags.go:64] FLAG: --healthz-port="10248" Sep 10 13:17:18.341945 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:17:18.335506 2557 flags.go:64] FLAG: --help="false" Sep 10 13:17:18.341945 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:17:18.335508 2557 flags.go:64] FLAG: --hostname-override="ip-10-0-139-174.ec2.internal" Sep 10 13:17:18.341945 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:17:18.335511 2557 flags.go:64] FLAG: --housekeeping-interval="10s" Sep 10 13:17:18.341945 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:17:18.335514 2557 flags.go:64] FLAG: --http-check-frequency="20s" Sep 10 13:17:18.341945 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:17:18.335516 2557 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins" Sep 10 13:17:18.341945 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:17:18.335519 2557 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml" Sep 10 13:17:18.341945 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:17:18.335523 2557 flags.go:64] FLAG: --image-gc-high-threshold="85" Sep 10 13:17:18.341945 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:17:18.335526 2557 flags.go:64] FLAG: --image-gc-low-threshold="80" Sep 10 13:17:18.341945 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:17:18.335528 2557 flags.go:64] FLAG: --image-service-endpoint="" Sep 10 13:17:18.342624 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:17:18.335531 2557 flags.go:64] FLAG: --kernel-memcg-notification="false" Sep 10 13:17:18.342624 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:17:18.335534 2557 flags.go:64] FLAG: --kube-api-burst="100" Sep 10 13:17:18.342624 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:17:18.335537 2557 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf" Sep 10 13:17:18.342624 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:17:18.335540 2557 flags.go:64] FLAG: --kube-api-qps="50" Sep 10 13:17:18.342624 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:17:18.335542 2557 flags.go:64] FLAG: --kube-reserved="" Sep 10 13:17:18.342624 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:17:18.335545 2557 flags.go:64] FLAG: --kube-reserved-cgroup="" Sep 10 13:17:18.342624 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:17:18.335549 2557 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig" Sep 10 13:17:18.342624 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:17:18.335551 2557 flags.go:64] FLAG: --kubelet-cgroups="" Sep 10 13:17:18.342624 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:17:18.335554 2557 flags.go:64] FLAG: --local-storage-capacity-isolation="true" Sep 10 13:17:18.342624 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:17:18.335557 2557 flags.go:64] FLAG: --lock-file="" Sep 10 13:17:18.342624 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:17:18.335560 2557 flags.go:64] FLAG: --log-cadvisor-usage="false" Sep 10 13:17:18.342624 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:17:18.335562 2557 flags.go:64] FLAG: --log-flush-frequency="5s" Sep 10 13:17:18.342624 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:17:18.335565 2557 flags.go:64] FLAG: --log-json-info-buffer-size="0" Sep 10 13:17:18.342624 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:17:18.335570 2557 flags.go:64] FLAG: --log-json-split-stream="false" Sep 10 13:17:18.342624 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:17:18.335573 2557 flags.go:64] FLAG: --log-text-info-buffer-size="0" Sep 10 13:17:18.342624 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:17:18.335576 2557 flags.go:64] FLAG: --log-text-split-stream="false" Sep 10 13:17:18.342624 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:17:18.335578 2557 flags.go:64] FLAG: --logging-format="text" Sep 10 13:17:18.342624 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:17:18.335581 2557 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id" Sep 10 13:17:18.342624 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:17:18.335584 2557 flags.go:64] FLAG: --make-iptables-util-chains="true" Sep 10 13:17:18.342624 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:17:18.335587 2557 flags.go:64] FLAG: --manifest-url="" Sep 10 13:17:18.342624 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:17:18.335590 2557 flags.go:64] FLAG: --manifest-url-header="" Sep 10 13:17:18.342624 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:17:18.335593 2557 flags.go:64] FLAG: --max-housekeeping-interval="15s" Sep 10 13:17:18.342624 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:17:18.335596 2557 flags.go:64] FLAG: --max-open-files="1000000" Sep 10 13:17:18.342624 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:17:18.335600 2557 flags.go:64] FLAG: --max-pods="110" Sep 10 13:17:18.342624 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:17:18.335603 2557 flags.go:64] FLAG: --maximum-dead-containers="-1" Sep 10 13:17:18.343245 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:17:18.335605 2557 flags.go:64] FLAG: --maximum-dead-containers-per-container="1" Sep 10 13:17:18.343245 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:17:18.335608 2557 flags.go:64] FLAG: --memory-manager-policy="None" Sep 10 13:17:18.343245 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:17:18.335611 2557 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s" Sep 10 13:17:18.343245 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:17:18.335614 2557 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s" Sep 10 13:17:18.343245 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:17:18.335617 2557 flags.go:64] FLAG: --node-ip="0.0.0.0" Sep 10 13:17:18.343245 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:17:18.335619 2557 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel" Sep 10 13:17:18.343245 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:17:18.335626 2557 flags.go:64] FLAG: --node-status-max-images="50" Sep 10 13:17:18.343245 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:17:18.335629 2557 flags.go:64] FLAG: --node-status-update-frequency="10s" Sep 10 13:17:18.343245 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:17:18.335631 2557 flags.go:64] FLAG: --oom-score-adj="-999" Sep 10 13:17:18.343245 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:17:18.335634 2557 flags.go:64] FLAG: --pod-cidr="" Sep 10 13:17:18.343245 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:17:18.335637 2557 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:8ccbacc999dbf9abad8f4009240188312b683f36ced4763eed166c1500c164b1" Sep 10 13:17:18.343245 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:17:18.335642 2557 flags.go:64] FLAG: --pod-manifest-path="" Sep 10 13:17:18.343245 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:17:18.335645 2557 flags.go:64] FLAG: --pod-max-pids="-1" Sep 10 13:17:18.343245 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:17:18.335649 2557 flags.go:64] FLAG: --pods-per-core="0" Sep 10 13:17:18.343245 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:17:18.335652 2557 flags.go:64] FLAG: --port="10250" Sep 10 13:17:18.343245 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:17:18.335654 2557 flags.go:64] FLAG: --protect-kernel-defaults="false" Sep 10 13:17:18.343245 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:17:18.335657 2557 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-00f973af1f5e9e3c2" Sep 10 13:17:18.343245 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:17:18.335660 2557 flags.go:64] FLAG: --qos-reserved="" Sep 10 13:17:18.343245 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:17:18.335663 2557 flags.go:64] FLAG: --read-only-port="10255" Sep 10 13:17:18.343245 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:17:18.335666 2557 flags.go:64] FLAG: --register-node="true" Sep 10 13:17:18.343245 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:17:18.335669 2557 flags.go:64] FLAG: --register-with-taints="" Sep 10 13:17:18.343245 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:17:18.335672 2557 flags.go:64] FLAG: --registry-burst="10" Sep 10 13:17:18.343245 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:17:18.335675 2557 flags.go:64] FLAG: --registry-qps="5" Sep 10 13:17:18.343245 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:17:18.335677 2557 flags.go:64] FLAG: --reserved-cpus="" Sep 10 13:17:18.343822 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:17:18.335680 2557 flags.go:64] FLAG: --reserved-memory="" Sep 10 13:17:18.343822 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:17:18.335683 2557 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf" Sep 10 13:17:18.343822 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:17:18.335686 2557 flags.go:64] FLAG: --root-dir="/var/lib/kubelet" Sep 10 13:17:18.343822 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:17:18.335689 2557 flags.go:64] FLAG: --rotate-certificates="false" Sep 10 13:17:18.343822 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:17:18.335692 2557 flags.go:64] FLAG: --rotate-server-certificates="false" Sep 10 13:17:18.343822 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:17:18.335694 2557 flags.go:64] FLAG: --runonce="false" Sep 10 13:17:18.343822 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:17:18.335697 2557 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service" Sep 10 13:17:18.343822 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:17:18.335700 2557 flags.go:64] FLAG: --runtime-request-timeout="2m0s" Sep 10 13:17:18.343822 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:17:18.335703 2557 flags.go:64] FLAG: --seccomp-default="false" Sep 10 13:17:18.343822 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:17:18.335705 2557 flags.go:64] FLAG: --serialize-image-pulls="true" Sep 10 13:17:18.343822 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:17:18.335708 2557 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s" Sep 10 13:17:18.343822 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:17:18.335711 2557 flags.go:64] FLAG: --storage-driver-db="cadvisor" Sep 10 13:17:18.343822 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:17:18.335714 2557 flags.go:64] FLAG: --storage-driver-host="localhost:8086" Sep 10 13:17:18.343822 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:17:18.335717 2557 flags.go:64] FLAG: --storage-driver-password="root" Sep 10 13:17:18.343822 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:17:18.335719 2557 flags.go:64] FLAG: --storage-driver-secure="false" Sep 10 13:17:18.343822 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:17:18.335722 2557 flags.go:64] FLAG: --storage-driver-table="stats" Sep 10 13:17:18.343822 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:17:18.335725 2557 flags.go:64] FLAG: --storage-driver-user="root" Sep 10 13:17:18.343822 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:17:18.335727 2557 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s" Sep 10 13:17:18.343822 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:17:18.335730 2557 flags.go:64] FLAG: --sync-frequency="1m0s" Sep 10 13:17:18.343822 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:17:18.335733 2557 flags.go:64] FLAG: --system-cgroups="" Sep 10 13:17:18.343822 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:17:18.335736 2557 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi" Sep 10 13:17:18.343822 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:17:18.335742 2557 flags.go:64] FLAG: --system-reserved-cgroup="" Sep 10 13:17:18.343822 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:17:18.335745 2557 flags.go:64] FLAG: --tls-cert-file="" Sep 10 13:17:18.343822 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:17:18.335748 2557 flags.go:64] FLAG: --tls-cipher-suites="[]" Sep 10 13:17:18.343822 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:17:18.335751 2557 flags.go:64] FLAG: --tls-min-version="" Sep 10 13:17:18.344460 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:17:18.335753 2557 flags.go:64] FLAG: --tls-private-key-file="" Sep 10 13:17:18.344460 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:17:18.335756 2557 flags.go:64] FLAG: --topology-manager-policy="none" Sep 10 13:17:18.344460 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:17:18.335759 2557 flags.go:64] FLAG: --topology-manager-policy-options="" Sep 10 13:17:18.344460 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:17:18.335762 2557 flags.go:64] FLAG: --topology-manager-scope="container" Sep 10 13:17:18.344460 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:17:18.335764 2557 flags.go:64] FLAG: --v="2" Sep 10 13:17:18.344460 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:17:18.335768 2557 flags.go:64] FLAG: --version="false" Sep 10 13:17:18.344460 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:17:18.335772 2557 flags.go:64] FLAG: --vmodule="" Sep 10 13:17:18.344460 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:17:18.335775 2557 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec" Sep 10 13:17:18.344460 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:17:18.335778 2557 flags.go:64] FLAG: --volume-stats-agg-period="1m0s" Sep 10 13:17:18.344460 ip-10-0-139-174 kubenswrapper[2557]: W0910 13:17:18.335898 2557 feature_gate.go:328] unrecognized feature gate: OSStreams Sep 10 13:17:18.344460 ip-10-0-139-174 kubenswrapper[2557]: W0910 13:17:18.335902 2557 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Sep 10 13:17:18.344460 ip-10-0-139-174 kubenswrapper[2557]: W0910 13:17:18.335905 2557 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Sep 10 13:17:18.344460 ip-10-0-139-174 kubenswrapper[2557]: W0910 13:17:18.335908 2557 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Sep 10 13:17:18.344460 ip-10-0-139-174 kubenswrapper[2557]: W0910 13:17:18.335912 2557 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Sep 10 13:17:18.344460 ip-10-0-139-174 kubenswrapper[2557]: W0910 13:17:18.335915 2557 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Sep 10 13:17:18.344460 ip-10-0-139-174 kubenswrapper[2557]: W0910 13:17:18.335918 2557 feature_gate.go:328] unrecognized feature gate: PinnedImages Sep 10 13:17:18.344460 ip-10-0-139-174 kubenswrapper[2557]: W0910 13:17:18.335921 2557 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Sep 10 13:17:18.344460 ip-10-0-139-174 kubenswrapper[2557]: W0910 13:17:18.335923 2557 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Sep 10 13:17:18.344460 ip-10-0-139-174 kubenswrapper[2557]: W0910 13:17:18.335926 2557 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Sep 10 13:17:18.344460 ip-10-0-139-174 kubenswrapper[2557]: W0910 13:17:18.335928 2557 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Sep 10 13:17:18.344460 ip-10-0-139-174 kubenswrapper[2557]: W0910 13:17:18.335931 2557 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Sep 10 13:17:18.344960 ip-10-0-139-174 kubenswrapper[2557]: W0910 13:17:18.335933 2557 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Sep 10 13:17:18.344960 ip-10-0-139-174 kubenswrapper[2557]: W0910 13:17:18.335936 2557 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Sep 10 13:17:18.344960 ip-10-0-139-174 kubenswrapper[2557]: W0910 13:17:18.335938 2557 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Sep 10 13:17:18.344960 ip-10-0-139-174 kubenswrapper[2557]: W0910 13:17:18.335940 2557 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Sep 10 13:17:18.344960 ip-10-0-139-174 kubenswrapper[2557]: W0910 13:17:18.335943 2557 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Sep 10 13:17:18.344960 ip-10-0-139-174 kubenswrapper[2557]: W0910 13:17:18.335945 2557 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Sep 10 13:17:18.344960 ip-10-0-139-174 kubenswrapper[2557]: W0910 13:17:18.335949 2557 feature_gate.go:328] unrecognized feature gate: InsightsConfig Sep 10 13:17:18.344960 ip-10-0-139-174 kubenswrapper[2557]: W0910 13:17:18.335952 2557 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Sep 10 13:17:18.344960 ip-10-0-139-174 kubenswrapper[2557]: W0910 13:17:18.335955 2557 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Sep 10 13:17:18.344960 ip-10-0-139-174 kubenswrapper[2557]: W0910 13:17:18.335957 2557 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Sep 10 13:17:18.344960 ip-10-0-139-174 kubenswrapper[2557]: W0910 13:17:18.335959 2557 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Sep 10 13:17:18.344960 ip-10-0-139-174 kubenswrapper[2557]: W0910 13:17:18.335961 2557 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Sep 10 13:17:18.344960 ip-10-0-139-174 kubenswrapper[2557]: W0910 13:17:18.335966 2557 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Sep 10 13:17:18.344960 ip-10-0-139-174 kubenswrapper[2557]: W0910 13:17:18.335968 2557 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Sep 10 13:17:18.344960 ip-10-0-139-174 kubenswrapper[2557]: W0910 13:17:18.335971 2557 feature_gate.go:328] unrecognized feature gate: OVNObservability Sep 10 13:17:18.344960 ip-10-0-139-174 kubenswrapper[2557]: W0910 13:17:18.335973 2557 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Sep 10 13:17:18.344960 ip-10-0-139-174 kubenswrapper[2557]: W0910 13:17:18.335975 2557 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Sep 10 13:17:18.344960 ip-10-0-139-174 kubenswrapper[2557]: W0910 13:17:18.335978 2557 feature_gate.go:328] unrecognized feature gate: SignatureStores Sep 10 13:17:18.344960 ip-10-0-139-174 kubenswrapper[2557]: W0910 13:17:18.335980 2557 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Sep 10 13:17:18.344960 ip-10-0-139-174 kubenswrapper[2557]: W0910 13:17:18.335983 2557 feature_gate.go:328] unrecognized feature gate: Example Sep 10 13:17:18.345459 ip-10-0-139-174 kubenswrapper[2557]: W0910 13:17:18.335985 2557 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Sep 10 13:17:18.345459 ip-10-0-139-174 kubenswrapper[2557]: W0910 13:17:18.335988 2557 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Sep 10 13:17:18.345459 ip-10-0-139-174 kubenswrapper[2557]: W0910 13:17:18.335990 2557 feature_gate.go:328] unrecognized feature gate: Example2 Sep 10 13:17:18.345459 ip-10-0-139-174 kubenswrapper[2557]: W0910 13:17:18.335993 2557 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Sep 10 13:17:18.345459 ip-10-0-139-174 kubenswrapper[2557]: W0910 13:17:18.335995 2557 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Sep 10 13:17:18.345459 ip-10-0-139-174 kubenswrapper[2557]: W0910 13:17:18.335999 2557 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Sep 10 13:17:18.345459 ip-10-0-139-174 kubenswrapper[2557]: W0910 13:17:18.336002 2557 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Sep 10 13:17:18.345459 ip-10-0-139-174 kubenswrapper[2557]: W0910 13:17:18.336005 2557 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Sep 10 13:17:18.345459 ip-10-0-139-174 kubenswrapper[2557]: W0910 13:17:18.336008 2557 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Sep 10 13:17:18.345459 ip-10-0-139-174 kubenswrapper[2557]: W0910 13:17:18.336010 2557 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Sep 10 13:17:18.345459 ip-10-0-139-174 kubenswrapper[2557]: W0910 13:17:18.336012 2557 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Sep 10 13:17:18.345459 ip-10-0-139-174 kubenswrapper[2557]: W0910 13:17:18.336015 2557 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Sep 10 13:17:18.345459 ip-10-0-139-174 kubenswrapper[2557]: W0910 13:17:18.336017 2557 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Sep 10 13:17:18.345459 ip-10-0-139-174 kubenswrapper[2557]: W0910 13:17:18.336020 2557 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Sep 10 13:17:18.345459 ip-10-0-139-174 kubenswrapper[2557]: W0910 13:17:18.336022 2557 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Sep 10 13:17:18.345459 ip-10-0-139-174 kubenswrapper[2557]: W0910 13:17:18.336024 2557 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Sep 10 13:17:18.345459 ip-10-0-139-174 kubenswrapper[2557]: W0910 13:17:18.336027 2557 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Sep 10 13:17:18.345459 ip-10-0-139-174 kubenswrapper[2557]: W0910 13:17:18.336029 2557 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Sep 10 13:17:18.345459 ip-10-0-139-174 kubenswrapper[2557]: W0910 13:17:18.336031 2557 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Sep 10 13:17:18.345459 ip-10-0-139-174 kubenswrapper[2557]: W0910 13:17:18.336036 2557 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Sep 10 13:17:18.345956 ip-10-0-139-174 kubenswrapper[2557]: W0910 13:17:18.336038 2557 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Sep 10 13:17:18.345956 ip-10-0-139-174 kubenswrapper[2557]: W0910 13:17:18.336040 2557 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Sep 10 13:17:18.345956 ip-10-0-139-174 kubenswrapper[2557]: W0910 13:17:18.336043 2557 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Sep 10 13:17:18.345956 ip-10-0-139-174 kubenswrapper[2557]: W0910 13:17:18.336046 2557 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Sep 10 13:17:18.345956 ip-10-0-139-174 kubenswrapper[2557]: W0910 13:17:18.336049 2557 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Sep 10 13:17:18.345956 ip-10-0-139-174 kubenswrapper[2557]: W0910 13:17:18.336052 2557 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Sep 10 13:17:18.345956 ip-10-0-139-174 kubenswrapper[2557]: W0910 13:17:18.336054 2557 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Sep 10 13:17:18.345956 ip-10-0-139-174 kubenswrapper[2557]: W0910 13:17:18.336056 2557 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Sep 10 13:17:18.345956 ip-10-0-139-174 kubenswrapper[2557]: W0910 13:17:18.336059 2557 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Sep 10 13:17:18.345956 ip-10-0-139-174 kubenswrapper[2557]: W0910 13:17:18.336076 2557 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Sep 10 13:17:18.345956 ip-10-0-139-174 kubenswrapper[2557]: W0910 13:17:18.336079 2557 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Sep 10 13:17:18.345956 ip-10-0-139-174 kubenswrapper[2557]: W0910 13:17:18.336081 2557 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Sep 10 13:17:18.345956 ip-10-0-139-174 kubenswrapper[2557]: W0910 13:17:18.336084 2557 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Sep 10 13:17:18.345956 ip-10-0-139-174 kubenswrapper[2557]: W0910 13:17:18.336086 2557 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Sep 10 13:17:18.345956 ip-10-0-139-174 kubenswrapper[2557]: W0910 13:17:18.336088 2557 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Sep 10 13:17:18.345956 ip-10-0-139-174 kubenswrapper[2557]: W0910 13:17:18.336091 2557 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Sep 10 13:17:18.345956 ip-10-0-139-174 kubenswrapper[2557]: W0910 13:17:18.336094 2557 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Sep 10 13:17:18.345956 ip-10-0-139-174 kubenswrapper[2557]: W0910 13:17:18.336096 2557 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Sep 10 13:17:18.345956 ip-10-0-139-174 kubenswrapper[2557]: W0910 13:17:18.336098 2557 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Sep 10 13:17:18.345956 ip-10-0-139-174 kubenswrapper[2557]: W0910 13:17:18.336101 2557 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Sep 10 13:17:18.346488 ip-10-0-139-174 kubenswrapper[2557]: W0910 13:17:18.336104 2557 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Sep 10 13:17:18.346488 ip-10-0-139-174 kubenswrapper[2557]: W0910 13:17:18.336106 2557 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Sep 10 13:17:18.346488 ip-10-0-139-174 kubenswrapper[2557]: W0910 13:17:18.336108 2557 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Sep 10 13:17:18.346488 ip-10-0-139-174 kubenswrapper[2557]: W0910 13:17:18.336111 2557 feature_gate.go:328] unrecognized feature gate: GatewayAPI Sep 10 13:17:18.346488 ip-10-0-139-174 kubenswrapper[2557]: W0910 13:17:18.336113 2557 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Sep 10 13:17:18.346488 ip-10-0-139-174 kubenswrapper[2557]: W0910 13:17:18.336116 2557 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Sep 10 13:17:18.346488 ip-10-0-139-174 kubenswrapper[2557]: W0910 13:17:18.336120 2557 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Sep 10 13:17:18.346488 ip-10-0-139-174 kubenswrapper[2557]: W0910 13:17:18.336123 2557 feature_gate.go:328] unrecognized feature gate: NewOLM Sep 10 13:17:18.346488 ip-10-0-139-174 kubenswrapper[2557]: W0910 13:17:18.336126 2557 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Sep 10 13:17:18.346488 ip-10-0-139-174 kubenswrapper[2557]: W0910 13:17:18.336128 2557 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Sep 10 13:17:18.346488 ip-10-0-139-174 kubenswrapper[2557]: W0910 13:17:18.336131 2557 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Sep 10 13:17:18.346488 ip-10-0-139-174 kubenswrapper[2557]: W0910 13:17:18.336135 2557 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Sep 10 13:17:18.346488 ip-10-0-139-174 kubenswrapper[2557]: W0910 13:17:18.336138 2557 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Sep 10 13:17:18.346488 ip-10-0-139-174 kubenswrapper[2557]: W0910 13:17:18.336141 2557 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Sep 10 13:17:18.346488 ip-10-0-139-174 kubenswrapper[2557]: W0910 13:17:18.336144 2557 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Sep 10 13:17:18.346488 ip-10-0-139-174 kubenswrapper[2557]: W0910 13:17:18.336147 2557 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Sep 10 13:17:18.346488 ip-10-0-139-174 kubenswrapper[2557]: W0910 13:17:18.336150 2557 feature_gate.go:328] unrecognized feature gate: DualReplica Sep 10 13:17:18.346488 ip-10-0-139-174 kubenswrapper[2557]: W0910 13:17:18.336153 2557 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Sep 10 13:17:18.346488 ip-10-0-139-174 kubenswrapper[2557]: W0910 13:17:18.336156 2557 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Sep 10 13:17:18.346956 ip-10-0-139-174 kubenswrapper[2557]: W0910 13:17:18.336158 2557 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Sep 10 13:17:18.346956 ip-10-0-139-174 kubenswrapper[2557]: W0910 13:17:18.336161 2557 feature_gate.go:328] unrecognized feature gate: KMSEncryption Sep 10 13:17:18.346956 ip-10-0-139-174 kubenswrapper[2557]: W0910 13:17:18.336164 2557 feature_gate.go:328] unrecognized feature gate: EventTTL Sep 10 13:17:18.346956 ip-10-0-139-174 kubenswrapper[2557]: W0910 13:17:18.336166 2557 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Sep 10 13:17:18.346956 ip-10-0-139-174 kubenswrapper[2557]: W0910 13:17:18.336169 2557 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Sep 10 13:17:18.346956 ip-10-0-139-174 kubenswrapper[2557]: W0910 13:17:18.336171 2557 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Sep 10 13:17:18.346956 ip-10-0-139-174 kubenswrapper[2557]: W0910 13:17:18.336174 2557 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Sep 10 13:17:18.346956 ip-10-0-139-174 kubenswrapper[2557]: W0910 13:17:18.336176 2557 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Sep 10 13:17:18.346956 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:17:18.337120 2557 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Sep 10 13:17:18.346956 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:17:18.343971 2557 server.go:529] "Kubelet version" kubeletVersion="v1.34.9" Sep 10 13:17:18.346956 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:17:18.344100 2557 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK="" Sep 10 13:17:18.346956 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:17:18.344816 2557 watchdog_linux.go:95] "Systemd watchdog is not enabled" Sep 10 13:17:18.346956 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:17:18.344823 2557 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started." Sep 10 13:17:18.346956 ip-10-0-139-174 kubenswrapper[2557]: W0910 13:17:18.344882 2557 feature_gate.go:328] unrecognized feature gate: OSStreams Sep 10 13:17:18.346956 ip-10-0-139-174 kubenswrapper[2557]: W0910 13:17:18.344886 2557 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Sep 10 13:17:18.347352 ip-10-0-139-174 kubenswrapper[2557]: W0910 13:17:18.344889 2557 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Sep 10 13:17:18.347352 ip-10-0-139-174 kubenswrapper[2557]: W0910 13:17:18.344893 2557 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Sep 10 13:17:18.347352 ip-10-0-139-174 kubenswrapper[2557]: W0910 13:17:18.344897 2557 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Sep 10 13:17:18.347352 ip-10-0-139-174 kubenswrapper[2557]: W0910 13:17:18.344901 2557 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Sep 10 13:17:18.347352 ip-10-0-139-174 kubenswrapper[2557]: W0910 13:17:18.344904 2557 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Sep 10 13:17:18.347352 ip-10-0-139-174 kubenswrapper[2557]: W0910 13:17:18.344907 2557 feature_gate.go:328] unrecognized feature gate: PinnedImages Sep 10 13:17:18.347352 ip-10-0-139-174 kubenswrapper[2557]: W0910 13:17:18.344910 2557 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Sep 10 13:17:18.347352 ip-10-0-139-174 kubenswrapper[2557]: W0910 13:17:18.344913 2557 feature_gate.go:328] unrecognized feature gate: SignatureStores Sep 10 13:17:18.347352 ip-10-0-139-174 kubenswrapper[2557]: W0910 13:17:18.344915 2557 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Sep 10 13:17:18.347352 ip-10-0-139-174 kubenswrapper[2557]: W0910 13:17:18.344918 2557 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Sep 10 13:17:18.347352 ip-10-0-139-174 kubenswrapper[2557]: W0910 13:17:18.344922 2557 feature_gate.go:328] unrecognized feature gate: InsightsConfig Sep 10 13:17:18.347352 ip-10-0-139-174 kubenswrapper[2557]: W0910 13:17:18.344924 2557 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Sep 10 13:17:18.347352 ip-10-0-139-174 kubenswrapper[2557]: W0910 13:17:18.344927 2557 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Sep 10 13:17:18.347352 ip-10-0-139-174 kubenswrapper[2557]: W0910 13:17:18.344929 2557 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Sep 10 13:17:18.347352 ip-10-0-139-174 kubenswrapper[2557]: W0910 13:17:18.344932 2557 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Sep 10 13:17:18.347352 ip-10-0-139-174 kubenswrapper[2557]: W0910 13:17:18.344934 2557 feature_gate.go:328] unrecognized feature gate: Example Sep 10 13:17:18.347352 ip-10-0-139-174 kubenswrapper[2557]: W0910 13:17:18.344938 2557 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Sep 10 13:17:18.347352 ip-10-0-139-174 kubenswrapper[2557]: W0910 13:17:18.344940 2557 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Sep 10 13:17:18.347352 ip-10-0-139-174 kubenswrapper[2557]: W0910 13:17:18.344944 2557 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Sep 10 13:17:18.347845 ip-10-0-139-174 kubenswrapper[2557]: W0910 13:17:18.344948 2557 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Sep 10 13:17:18.347845 ip-10-0-139-174 kubenswrapper[2557]: W0910 13:17:18.344951 2557 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Sep 10 13:17:18.347845 ip-10-0-139-174 kubenswrapper[2557]: W0910 13:17:18.344954 2557 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Sep 10 13:17:18.347845 ip-10-0-139-174 kubenswrapper[2557]: W0910 13:17:18.344957 2557 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Sep 10 13:17:18.347845 ip-10-0-139-174 kubenswrapper[2557]: W0910 13:17:18.344960 2557 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Sep 10 13:17:18.347845 ip-10-0-139-174 kubenswrapper[2557]: W0910 13:17:18.344963 2557 feature_gate.go:328] unrecognized feature gate: EventTTL Sep 10 13:17:18.347845 ip-10-0-139-174 kubenswrapper[2557]: W0910 13:17:18.344966 2557 feature_gate.go:328] unrecognized feature gate: NewOLM Sep 10 13:17:18.347845 ip-10-0-139-174 kubenswrapper[2557]: W0910 13:17:18.344968 2557 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Sep 10 13:17:18.347845 ip-10-0-139-174 kubenswrapper[2557]: W0910 13:17:18.344972 2557 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Sep 10 13:17:18.347845 ip-10-0-139-174 kubenswrapper[2557]: W0910 13:17:18.344975 2557 feature_gate.go:328] unrecognized feature gate: GatewayAPI Sep 10 13:17:18.347845 ip-10-0-139-174 kubenswrapper[2557]: W0910 13:17:18.344978 2557 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Sep 10 13:17:18.347845 ip-10-0-139-174 kubenswrapper[2557]: W0910 13:17:18.344980 2557 feature_gate.go:328] unrecognized feature gate: OVNObservability Sep 10 13:17:18.347845 ip-10-0-139-174 kubenswrapper[2557]: W0910 13:17:18.344983 2557 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Sep 10 13:17:18.347845 ip-10-0-139-174 kubenswrapper[2557]: W0910 13:17:18.344985 2557 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Sep 10 13:17:18.347845 ip-10-0-139-174 kubenswrapper[2557]: W0910 13:17:18.344988 2557 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Sep 10 13:17:18.347845 ip-10-0-139-174 kubenswrapper[2557]: W0910 13:17:18.344990 2557 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Sep 10 13:17:18.347845 ip-10-0-139-174 kubenswrapper[2557]: W0910 13:17:18.344992 2557 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Sep 10 13:17:18.347845 ip-10-0-139-174 kubenswrapper[2557]: W0910 13:17:18.344995 2557 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Sep 10 13:17:18.347845 ip-10-0-139-174 kubenswrapper[2557]: W0910 13:17:18.344997 2557 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Sep 10 13:17:18.348340 ip-10-0-139-174 kubenswrapper[2557]: W0910 13:17:18.345000 2557 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Sep 10 13:17:18.348340 ip-10-0-139-174 kubenswrapper[2557]: W0910 13:17:18.345002 2557 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Sep 10 13:17:18.348340 ip-10-0-139-174 kubenswrapper[2557]: W0910 13:17:18.345005 2557 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Sep 10 13:17:18.348340 ip-10-0-139-174 kubenswrapper[2557]: W0910 13:17:18.345007 2557 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Sep 10 13:17:18.348340 ip-10-0-139-174 kubenswrapper[2557]: W0910 13:17:18.345009 2557 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Sep 10 13:17:18.348340 ip-10-0-139-174 kubenswrapper[2557]: W0910 13:17:18.345012 2557 feature_gate.go:328] unrecognized feature gate: Example2 Sep 10 13:17:18.348340 ip-10-0-139-174 kubenswrapper[2557]: W0910 13:17:18.345015 2557 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Sep 10 13:17:18.348340 ip-10-0-139-174 kubenswrapper[2557]: W0910 13:17:18.345017 2557 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Sep 10 13:17:18.348340 ip-10-0-139-174 kubenswrapper[2557]: W0910 13:17:18.345019 2557 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Sep 10 13:17:18.348340 ip-10-0-139-174 kubenswrapper[2557]: W0910 13:17:18.345022 2557 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Sep 10 13:17:18.348340 ip-10-0-139-174 kubenswrapper[2557]: W0910 13:17:18.345024 2557 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Sep 10 13:17:18.348340 ip-10-0-139-174 kubenswrapper[2557]: W0910 13:17:18.345026 2557 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Sep 10 13:17:18.348340 ip-10-0-139-174 kubenswrapper[2557]: W0910 13:17:18.345030 2557 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Sep 10 13:17:18.348340 ip-10-0-139-174 kubenswrapper[2557]: W0910 13:17:18.345032 2557 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Sep 10 13:17:18.348340 ip-10-0-139-174 kubenswrapper[2557]: W0910 13:17:18.345035 2557 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Sep 10 13:17:18.348340 ip-10-0-139-174 kubenswrapper[2557]: W0910 13:17:18.345039 2557 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Sep 10 13:17:18.348340 ip-10-0-139-174 kubenswrapper[2557]: W0910 13:17:18.345041 2557 feature_gate.go:328] unrecognized feature gate: DualReplica Sep 10 13:17:18.348340 ip-10-0-139-174 kubenswrapper[2557]: W0910 13:17:18.345044 2557 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Sep 10 13:17:18.348340 ip-10-0-139-174 kubenswrapper[2557]: W0910 13:17:18.345046 2557 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Sep 10 13:17:18.348340 ip-10-0-139-174 kubenswrapper[2557]: W0910 13:17:18.345049 2557 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Sep 10 13:17:18.348818 ip-10-0-139-174 kubenswrapper[2557]: W0910 13:17:18.345051 2557 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Sep 10 13:17:18.348818 ip-10-0-139-174 kubenswrapper[2557]: W0910 13:17:18.345054 2557 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Sep 10 13:17:18.348818 ip-10-0-139-174 kubenswrapper[2557]: W0910 13:17:18.345056 2557 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Sep 10 13:17:18.348818 ip-10-0-139-174 kubenswrapper[2557]: W0910 13:17:18.345059 2557 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Sep 10 13:17:18.348818 ip-10-0-139-174 kubenswrapper[2557]: W0910 13:17:18.345080 2557 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Sep 10 13:17:18.348818 ip-10-0-139-174 kubenswrapper[2557]: W0910 13:17:18.345084 2557 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Sep 10 13:17:18.348818 ip-10-0-139-174 kubenswrapper[2557]: W0910 13:17:18.345087 2557 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Sep 10 13:17:18.348818 ip-10-0-139-174 kubenswrapper[2557]: W0910 13:17:18.345089 2557 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Sep 10 13:17:18.348818 ip-10-0-139-174 kubenswrapper[2557]: W0910 13:17:18.345092 2557 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Sep 10 13:17:18.348818 ip-10-0-139-174 kubenswrapper[2557]: W0910 13:17:18.345094 2557 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Sep 10 13:17:18.348818 ip-10-0-139-174 kubenswrapper[2557]: W0910 13:17:18.345097 2557 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Sep 10 13:17:18.348818 ip-10-0-139-174 kubenswrapper[2557]: W0910 13:17:18.345099 2557 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Sep 10 13:17:18.348818 ip-10-0-139-174 kubenswrapper[2557]: W0910 13:17:18.345102 2557 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Sep 10 13:17:18.348818 ip-10-0-139-174 kubenswrapper[2557]: W0910 13:17:18.345104 2557 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Sep 10 13:17:18.348818 ip-10-0-139-174 kubenswrapper[2557]: W0910 13:17:18.345106 2557 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Sep 10 13:17:18.348818 ip-10-0-139-174 kubenswrapper[2557]: W0910 13:17:18.345109 2557 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Sep 10 13:17:18.348818 ip-10-0-139-174 kubenswrapper[2557]: W0910 13:17:18.345111 2557 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Sep 10 13:17:18.348818 ip-10-0-139-174 kubenswrapper[2557]: W0910 13:17:18.345114 2557 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Sep 10 13:17:18.348818 ip-10-0-139-174 kubenswrapper[2557]: W0910 13:17:18.345117 2557 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Sep 10 13:17:18.348818 ip-10-0-139-174 kubenswrapper[2557]: W0910 13:17:18.345119 2557 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Sep 10 13:17:18.349363 ip-10-0-139-174 kubenswrapper[2557]: W0910 13:17:18.345122 2557 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Sep 10 13:17:18.349363 ip-10-0-139-174 kubenswrapper[2557]: W0910 13:17:18.345124 2557 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Sep 10 13:17:18.349363 ip-10-0-139-174 kubenswrapper[2557]: W0910 13:17:18.345127 2557 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Sep 10 13:17:18.349363 ip-10-0-139-174 kubenswrapper[2557]: W0910 13:17:18.345129 2557 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Sep 10 13:17:18.349363 ip-10-0-139-174 kubenswrapper[2557]: W0910 13:17:18.345132 2557 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Sep 10 13:17:18.349363 ip-10-0-139-174 kubenswrapper[2557]: W0910 13:17:18.345135 2557 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Sep 10 13:17:18.349363 ip-10-0-139-174 kubenswrapper[2557]: W0910 13:17:18.345137 2557 feature_gate.go:328] unrecognized feature gate: KMSEncryption Sep 10 13:17:18.349363 ip-10-0-139-174 kubenswrapper[2557]: W0910 13:17:18.345142 2557 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Sep 10 13:17:18.349363 ip-10-0-139-174 kubenswrapper[2557]: W0910 13:17:18.345145 2557 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Sep 10 13:17:18.349363 ip-10-0-139-174 kubenswrapper[2557]: W0910 13:17:18.345148 2557 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Sep 10 13:17:18.349363 ip-10-0-139-174 kubenswrapper[2557]: W0910 13:17:18.345151 2557 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Sep 10 13:17:18.349363 ip-10-0-139-174 kubenswrapper[2557]: W0910 13:17:18.345154 2557 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Sep 10 13:17:18.349363 ip-10-0-139-174 kubenswrapper[2557]: W0910 13:17:18.345156 2557 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Sep 10 13:17:18.349363 ip-10-0-139-174 kubenswrapper[2557]: W0910 13:17:18.345159 2557 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Sep 10 13:17:18.349363 ip-10-0-139-174 kubenswrapper[2557]: W0910 13:17:18.345161 2557 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Sep 10 13:17:18.349363 ip-10-0-139-174 kubenswrapper[2557]: W0910 13:17:18.345164 2557 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Sep 10 13:17:18.349363 ip-10-0-139-174 kubenswrapper[2557]: W0910 13:17:18.345166 2557 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Sep 10 13:17:18.349363 ip-10-0-139-174 kubenswrapper[2557]: W0910 13:17:18.345169 2557 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Sep 10 13:17:18.349363 ip-10-0-139-174 kubenswrapper[2557]: W0910 13:17:18.345171 2557 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Sep 10 13:17:18.349822 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:17:18.345176 2557 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Sep 10 13:17:18.349822 ip-10-0-139-174 kubenswrapper[2557]: W0910 13:17:18.345285 2557 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Sep 10 13:17:18.349822 ip-10-0-139-174 kubenswrapper[2557]: W0910 13:17:18.345289 2557 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Sep 10 13:17:18.349822 ip-10-0-139-174 kubenswrapper[2557]: W0910 13:17:18.345292 2557 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Sep 10 13:17:18.349822 ip-10-0-139-174 kubenswrapper[2557]: W0910 13:17:18.345295 2557 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Sep 10 13:17:18.349822 ip-10-0-139-174 kubenswrapper[2557]: W0910 13:17:18.345297 2557 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Sep 10 13:17:18.349822 ip-10-0-139-174 kubenswrapper[2557]: W0910 13:17:18.345300 2557 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Sep 10 13:17:18.349822 ip-10-0-139-174 kubenswrapper[2557]: W0910 13:17:18.345302 2557 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Sep 10 13:17:18.349822 ip-10-0-139-174 kubenswrapper[2557]: W0910 13:17:18.345305 2557 feature_gate.go:328] unrecognized feature gate: OSStreams Sep 10 13:17:18.349822 ip-10-0-139-174 kubenswrapper[2557]: W0910 13:17:18.345307 2557 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Sep 10 13:17:18.349822 ip-10-0-139-174 kubenswrapper[2557]: W0910 13:17:18.345309 2557 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Sep 10 13:17:18.349822 ip-10-0-139-174 kubenswrapper[2557]: W0910 13:17:18.345312 2557 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Sep 10 13:17:18.349822 ip-10-0-139-174 kubenswrapper[2557]: W0910 13:17:18.345315 2557 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Sep 10 13:17:18.349822 ip-10-0-139-174 kubenswrapper[2557]: W0910 13:17:18.345317 2557 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Sep 10 13:17:18.349822 ip-10-0-139-174 kubenswrapper[2557]: W0910 13:17:18.345320 2557 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Sep 10 13:17:18.350217 ip-10-0-139-174 kubenswrapper[2557]: W0910 13:17:18.345322 2557 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Sep 10 13:17:18.350217 ip-10-0-139-174 kubenswrapper[2557]: W0910 13:17:18.345325 2557 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Sep 10 13:17:18.350217 ip-10-0-139-174 kubenswrapper[2557]: W0910 13:17:18.345328 2557 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Sep 10 13:17:18.350217 ip-10-0-139-174 kubenswrapper[2557]: W0910 13:17:18.345330 2557 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Sep 10 13:17:18.350217 ip-10-0-139-174 kubenswrapper[2557]: W0910 13:17:18.345333 2557 feature_gate.go:328] unrecognized feature gate: GatewayAPI Sep 10 13:17:18.350217 ip-10-0-139-174 kubenswrapper[2557]: W0910 13:17:18.345335 2557 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Sep 10 13:17:18.350217 ip-10-0-139-174 kubenswrapper[2557]: W0910 13:17:18.345338 2557 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Sep 10 13:17:18.350217 ip-10-0-139-174 kubenswrapper[2557]: W0910 13:17:18.345340 2557 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Sep 10 13:17:18.350217 ip-10-0-139-174 kubenswrapper[2557]: W0910 13:17:18.345343 2557 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Sep 10 13:17:18.350217 ip-10-0-139-174 kubenswrapper[2557]: W0910 13:17:18.345347 2557 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Sep 10 13:17:18.350217 ip-10-0-139-174 kubenswrapper[2557]: W0910 13:17:18.345350 2557 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Sep 10 13:17:18.350217 ip-10-0-139-174 kubenswrapper[2557]: W0910 13:17:18.345353 2557 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Sep 10 13:17:18.350217 ip-10-0-139-174 kubenswrapper[2557]: W0910 13:17:18.345357 2557 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Sep 10 13:17:18.350217 ip-10-0-139-174 kubenswrapper[2557]: W0910 13:17:18.345362 2557 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Sep 10 13:17:18.350217 ip-10-0-139-174 kubenswrapper[2557]: W0910 13:17:18.345366 2557 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Sep 10 13:17:18.350217 ip-10-0-139-174 kubenswrapper[2557]: W0910 13:17:18.345369 2557 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Sep 10 13:17:18.350217 ip-10-0-139-174 kubenswrapper[2557]: W0910 13:17:18.345373 2557 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Sep 10 13:17:18.350217 ip-10-0-139-174 kubenswrapper[2557]: W0910 13:17:18.345376 2557 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Sep 10 13:17:18.350217 ip-10-0-139-174 kubenswrapper[2557]: W0910 13:17:18.345378 2557 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Sep 10 13:17:18.350217 ip-10-0-139-174 kubenswrapper[2557]: W0910 13:17:18.345381 2557 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Sep 10 13:17:18.350703 ip-10-0-139-174 kubenswrapper[2557]: W0910 13:17:18.345383 2557 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Sep 10 13:17:18.350703 ip-10-0-139-174 kubenswrapper[2557]: W0910 13:17:18.345386 2557 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Sep 10 13:17:18.350703 ip-10-0-139-174 kubenswrapper[2557]: W0910 13:17:18.345388 2557 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Sep 10 13:17:18.350703 ip-10-0-139-174 kubenswrapper[2557]: W0910 13:17:18.345391 2557 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Sep 10 13:17:18.350703 ip-10-0-139-174 kubenswrapper[2557]: W0910 13:17:18.345393 2557 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Sep 10 13:17:18.350703 ip-10-0-139-174 kubenswrapper[2557]: W0910 13:17:18.345396 2557 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Sep 10 13:17:18.350703 ip-10-0-139-174 kubenswrapper[2557]: W0910 13:17:18.345398 2557 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Sep 10 13:17:18.350703 ip-10-0-139-174 kubenswrapper[2557]: W0910 13:17:18.345400 2557 feature_gate.go:328] unrecognized feature gate: InsightsConfig Sep 10 13:17:18.350703 ip-10-0-139-174 kubenswrapper[2557]: W0910 13:17:18.345403 2557 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Sep 10 13:17:18.350703 ip-10-0-139-174 kubenswrapper[2557]: W0910 13:17:18.345405 2557 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Sep 10 13:17:18.350703 ip-10-0-139-174 kubenswrapper[2557]: W0910 13:17:18.345409 2557 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Sep 10 13:17:18.350703 ip-10-0-139-174 kubenswrapper[2557]: W0910 13:17:18.345412 2557 feature_gate.go:328] unrecognized feature gate: Example2 Sep 10 13:17:18.350703 ip-10-0-139-174 kubenswrapper[2557]: W0910 13:17:18.345415 2557 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Sep 10 13:17:18.350703 ip-10-0-139-174 kubenswrapper[2557]: W0910 13:17:18.345417 2557 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Sep 10 13:17:18.350703 ip-10-0-139-174 kubenswrapper[2557]: W0910 13:17:18.345419 2557 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Sep 10 13:17:18.350703 ip-10-0-139-174 kubenswrapper[2557]: W0910 13:17:18.345422 2557 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Sep 10 13:17:18.350703 ip-10-0-139-174 kubenswrapper[2557]: W0910 13:17:18.345424 2557 feature_gate.go:328] unrecognized feature gate: OVNObservability Sep 10 13:17:18.350703 ip-10-0-139-174 kubenswrapper[2557]: W0910 13:17:18.345427 2557 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Sep 10 13:17:18.350703 ip-10-0-139-174 kubenswrapper[2557]: W0910 13:17:18.345429 2557 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Sep 10 13:17:18.351166 ip-10-0-139-174 kubenswrapper[2557]: W0910 13:17:18.345432 2557 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Sep 10 13:17:18.351166 ip-10-0-139-174 kubenswrapper[2557]: W0910 13:17:18.345434 2557 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Sep 10 13:17:18.351166 ip-10-0-139-174 kubenswrapper[2557]: W0910 13:17:18.345436 2557 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Sep 10 13:17:18.351166 ip-10-0-139-174 kubenswrapper[2557]: W0910 13:17:18.345439 2557 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Sep 10 13:17:18.351166 ip-10-0-139-174 kubenswrapper[2557]: W0910 13:17:18.345442 2557 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Sep 10 13:17:18.351166 ip-10-0-139-174 kubenswrapper[2557]: W0910 13:17:18.345444 2557 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Sep 10 13:17:18.351166 ip-10-0-139-174 kubenswrapper[2557]: W0910 13:17:18.345446 2557 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Sep 10 13:17:18.351166 ip-10-0-139-174 kubenswrapper[2557]: W0910 13:17:18.345449 2557 feature_gate.go:328] unrecognized feature gate: SignatureStores Sep 10 13:17:18.351166 ip-10-0-139-174 kubenswrapper[2557]: W0910 13:17:18.345451 2557 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Sep 10 13:17:18.351166 ip-10-0-139-174 kubenswrapper[2557]: W0910 13:17:18.345454 2557 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Sep 10 13:17:18.351166 ip-10-0-139-174 kubenswrapper[2557]: W0910 13:17:18.345456 2557 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Sep 10 13:17:18.351166 ip-10-0-139-174 kubenswrapper[2557]: W0910 13:17:18.345459 2557 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Sep 10 13:17:18.351166 ip-10-0-139-174 kubenswrapper[2557]: W0910 13:17:18.345461 2557 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Sep 10 13:17:18.351166 ip-10-0-139-174 kubenswrapper[2557]: W0910 13:17:18.345464 2557 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Sep 10 13:17:18.351166 ip-10-0-139-174 kubenswrapper[2557]: W0910 13:17:18.345467 2557 feature_gate.go:328] unrecognized feature gate: PinnedImages Sep 10 13:17:18.351166 ip-10-0-139-174 kubenswrapper[2557]: W0910 13:17:18.345469 2557 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Sep 10 13:17:18.351166 ip-10-0-139-174 kubenswrapper[2557]: W0910 13:17:18.345471 2557 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Sep 10 13:17:18.351166 ip-10-0-139-174 kubenswrapper[2557]: W0910 13:17:18.345474 2557 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Sep 10 13:17:18.351166 ip-10-0-139-174 kubenswrapper[2557]: W0910 13:17:18.345476 2557 feature_gate.go:328] unrecognized feature gate: DualReplica Sep 10 13:17:18.351166 ip-10-0-139-174 kubenswrapper[2557]: W0910 13:17:18.345479 2557 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Sep 10 13:17:18.351641 ip-10-0-139-174 kubenswrapper[2557]: W0910 13:17:18.345481 2557 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Sep 10 13:17:18.351641 ip-10-0-139-174 kubenswrapper[2557]: W0910 13:17:18.345483 2557 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Sep 10 13:17:18.351641 ip-10-0-139-174 kubenswrapper[2557]: W0910 13:17:18.345486 2557 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Sep 10 13:17:18.351641 ip-10-0-139-174 kubenswrapper[2557]: W0910 13:17:18.345488 2557 feature_gate.go:328] unrecognized feature gate: KMSEncryption Sep 10 13:17:18.351641 ip-10-0-139-174 kubenswrapper[2557]: W0910 13:17:18.345491 2557 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Sep 10 13:17:18.351641 ip-10-0-139-174 kubenswrapper[2557]: W0910 13:17:18.345493 2557 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Sep 10 13:17:18.351641 ip-10-0-139-174 kubenswrapper[2557]: W0910 13:17:18.345496 2557 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Sep 10 13:17:18.351641 ip-10-0-139-174 kubenswrapper[2557]: W0910 13:17:18.345498 2557 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Sep 10 13:17:18.351641 ip-10-0-139-174 kubenswrapper[2557]: W0910 13:17:18.345501 2557 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Sep 10 13:17:18.351641 ip-10-0-139-174 kubenswrapper[2557]: W0910 13:17:18.345504 2557 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Sep 10 13:17:18.351641 ip-10-0-139-174 kubenswrapper[2557]: W0910 13:17:18.345507 2557 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Sep 10 13:17:18.351641 ip-10-0-139-174 kubenswrapper[2557]: W0910 13:17:18.345511 2557 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Sep 10 13:17:18.351641 ip-10-0-139-174 kubenswrapper[2557]: W0910 13:17:18.345514 2557 feature_gate.go:328] unrecognized feature gate: EventTTL Sep 10 13:17:18.351641 ip-10-0-139-174 kubenswrapper[2557]: W0910 13:17:18.345517 2557 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Sep 10 13:17:18.351641 ip-10-0-139-174 kubenswrapper[2557]: W0910 13:17:18.345520 2557 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Sep 10 13:17:18.351641 ip-10-0-139-174 kubenswrapper[2557]: W0910 13:17:18.345523 2557 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Sep 10 13:17:18.351641 ip-10-0-139-174 kubenswrapper[2557]: W0910 13:17:18.345525 2557 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Sep 10 13:17:18.351641 ip-10-0-139-174 kubenswrapper[2557]: W0910 13:17:18.345528 2557 feature_gate.go:328] unrecognized feature gate: NewOLM Sep 10 13:17:18.351641 ip-10-0-139-174 kubenswrapper[2557]: W0910 13:17:18.345530 2557 feature_gate.go:328] unrecognized feature gate: Example Sep 10 13:17:18.352110 ip-10-0-139-174 kubenswrapper[2557]: W0910 13:17:18.345533 2557 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Sep 10 13:17:18.352110 ip-10-0-139-174 kubenswrapper[2557]: W0910 13:17:18.345535 2557 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Sep 10 13:17:18.352110 ip-10-0-139-174 kubenswrapper[2557]: W0910 13:17:18.345538 2557 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Sep 10 13:17:18.352110 ip-10-0-139-174 kubenswrapper[2557]: W0910 13:17:18.345540 2557 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Sep 10 13:17:18.352110 ip-10-0-139-174 kubenswrapper[2557]: W0910 13:17:18.345542 2557 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Sep 10 13:17:18.352110 ip-10-0-139-174 kubenswrapper[2557]: W0910 13:17:18.345545 2557 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Sep 10 13:17:18.352110 ip-10-0-139-174 kubenswrapper[2557]: W0910 13:17:18.345547 2557 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Sep 10 13:17:18.352110 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:17:18.345551 2557 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Sep 10 13:17:18.352110 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:17:18.345649 2557 server.go:962] "Client rotation is on, will bootstrap in background" Sep 10 13:17:18.352110 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:17:18.347734 2557 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir" Sep 10 13:17:18.352110 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:17:18.348990 2557 server.go:1019] "Starting client certificate rotation" Sep 10 13:17:18.352110 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:17:18.349096 2557 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet" Sep 10 13:17:18.352110 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:17:18.349142 2557 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet" Sep 10 13:17:18.353394 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:17:18.353353 2557 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Sep 10 13:17:18.354890 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:17:18.354870 2557 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Sep 10 13:17:18.367292 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:17:18.367269 2557 log.go:25] "Validated CRI v1 runtime API" Sep 10 13:17:18.371575 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:17:18.371540 2557 log.go:25] "Validated CRI v1 image API" Sep 10 13:17:18.372881 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:17:18.372865 2557 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd" Sep 10 13:17:18.375496 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:17:18.375477 2557 fs.go:135] Filesystem UUIDs: map[134b8f41-b5fc-46bf-85b9-3587a1e0f996:/dev/nvme0n1p4 7B77-95E7:/dev/nvme0n1p2 91ccfcdd-826c-41eb-b7a4-8f594444b01e:/dev/nvme0n1p3] Sep 10 13:17:18.375548 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:17:18.375496 2557 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}] Sep 10 13:17:18.380548 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:17:18.380444 2557 manager.go:216] Machine: {Timestamp:2026-09-10 13:17:18.379180674 +0000 UTC m=+0.335529154 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3098296 MemoryCapacity:32812163072 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec26710b67c9d22d05aed6dab2ce8e53 SystemUUID:ec26710b-67c9-d22d-05ae-d6dab2ce8e53 BootID:c8c51370-7b73-442a-8148-5aef3a1bb8b6 Filesystems:[{Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6562435072 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6111232 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16406081536 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16406081536 Type:vfs Inodes:4005391 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:2e:09:00:83:5d Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:2e:09:00:83:5d Speed:0 Mtu:9001} {Name:ovs-system MacAddress:0e:c1:50:c7:25:c1 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:32812163072 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:34603008 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None} Sep 10 13:17:18.380548 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:17:18.380546 2557 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available. Sep 10 13:17:18.380657 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:17:18.380641 2557 manager.go:232] Version: {KernelVersion:5.14.0-570.138.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260901-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:} Sep 10 13:17:18.382920 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:17:18.382897 2557 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[] Sep 10 13:17:18.383059 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:17:18.382924 2557 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-139-174.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2} Sep 10 13:17:18.383595 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:17:18.383580 2557 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Sep 10 13:17:18.383794 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:17:18.383783 2557 topology_manager.go:138] "Creating topology manager with none policy" Sep 10 13:17:18.383868 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:17:18.383797 2557 container_manager_linux.go:309] "Creating device plugin manager" Sep 10 13:17:18.383868 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:17:18.383805 2557 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock" Sep 10 13:17:18.385220 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:17:18.385209 2557 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock" Sep 10 13:17:18.386690 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:17:18.386680 2557 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager" Sep 10 13:17:18.390036 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:17:18.390025 2557 state_mem.go:36] "Initialized new in-memory state store" Sep 10 13:17:18.390171 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:17:18.390161 2557 server.go:1240] "Using root directory" path="/var/lib/kubelet" Sep 10 13:17:18.390200 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:17:18.390193 2557 kubelet.go:486] "Attempting to sync node with API server" Sep 10 13:17:18.390233 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:17:18.390212 2557 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests" Sep 10 13:17:18.390233 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:17:18.390226 2557 file.go:69] "Watching path" path="/etc/kubernetes/manifests" Sep 10 13:17:18.390284 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:17:18.390235 2557 kubelet.go:398] "Adding apiserver pod source" Sep 10 13:17:18.390284 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:17:18.390243 2557 apiserver.go:42] "Waiting for node sync before watching apiserver pods" Sep 10 13:17:18.390587 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:17:18.390574 2557 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Sep 10 13:17:18.390645 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:17:18.390594 2557 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Sep 10 13:17:18.396648 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:17:18.396619 2557 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.12-5.rhaos4.21.gitd1ec63c.el9" apiVersion="v1" Sep 10 13:17:18.399331 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:17:18.399315 2557 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled" Sep 10 13:17:18.399392 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:17:18.399348 2557 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled" Sep 10 13:17:18.399539 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:17:18.399528 2557 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume" Sep 10 13:17:18.399571 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:17:18.399545 2557 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir" Sep 10 13:17:18.399571 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:17:18.399554 2557 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo" Sep 10 13:17:18.399571 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:17:18.399563 2557 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path" Sep 10 13:17:18.399650 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:17:18.399575 2557 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs" Sep 10 13:17:18.399650 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:17:18.399581 2557 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret" Sep 10 13:17:18.400124 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:17:18.400114 2557 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi" Sep 10 13:17:18.400124 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:17:18.400124 2557 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api" Sep 10 13:17:18.400189 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:17:18.400131 2557 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc" Sep 10 13:17:18.400189 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:17:18.400136 2557 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap" Sep 10 13:17:18.400189 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:17:18.400145 2557 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected" Sep 10 13:17:18.400189 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:17:18.400153 2557 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume" Sep 10 13:17:18.400934 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:17:18.400924 2557 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi" Sep 10 13:17:18.400962 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:17:18.400936 2557 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image" Sep 10 13:17:18.404786 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:17:18.404772 2557 server.go:1268] "Started kubelet" Sep 10 13:17:18.404842 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:17:18.404819 2557 server.go:180] "Starting to listen" address="0.0.0.0" port=10250 Sep 10 13:17:18.404895 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:17:18.404845 2557 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10 Sep 10 13:17:18.404943 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:17:18.404909 2557 server_v1.go:49] "podresources" method="list" useActivePods=true Sep 10 13:17:18.405548 ip-10-0-139-174 systemd[1]: Started Kubernetes Kubelet. Sep 10 13:17:18.406365 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:17:18.406229 2557 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock" Sep 10 13:17:18.407048 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:17:18.407037 2557 server.go:310] "Adding debug handlers to kubelet server" Sep 10 13:17:18.409356 ip-10-0-139-174 kubenswrapper[2557]: E0910 13:17:18.409334 2557 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache" Sep 10 13:17:18.413114 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:17:18.413100 2557 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer" Sep 10 13:17:18.413227 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:17:18.413205 2557 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving" Sep 10 13:17:18.413380 ip-10-0-139-174 kubenswrapper[2557]: E0910 13:17:18.413220 2557 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-139-174.ec2.internal\" not found" Sep 10 13:17:18.413882 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:17:18.413857 2557 volume_manager.go:311] "The desired_state_of_world populator starts" Sep 10 13:17:18.413882 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:17:18.413884 2557 volume_manager.go:313] "Starting Kubelet Volume Manager" Sep 10 13:17:18.414002 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:17:18.413870 2557 desired_state_of_world_populator.go:146] "Desired state populator starts to run" Sep 10 13:17:18.414002 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:17:18.413896 2557 factory.go:55] Registering systemd factory Sep 10 13:17:18.414002 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:17:18.413912 2557 factory.go:223] Registration of the systemd container factory successfully Sep 10 13:17:18.414139 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:17:18.414014 2557 reconstruct.go:97] "Volume reconstruction finished" Sep 10 13:17:18.414139 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:17:18.414024 2557 reconciler.go:29] "Reconciler: start to sync state" Sep 10 13:17:18.414279 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:17:18.414266 2557 factory.go:153] Registering CRI-O factory Sep 10 13:17:18.414324 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:17:18.414285 2557 factory.go:223] Registration of the crio container factory successfully Sep 10 13:17:18.414552 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:17:18.414338 2557 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory Sep 10 13:17:18.414552 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:17:18.414368 2557 factory.go:103] Registering Raw factory Sep 10 13:17:18.414552 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:17:18.414385 2557 manager.go:1196] Started watching for new ooms in manager Sep 10 13:17:18.414913 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:17:18.414807 2557 manager.go:349] Starting recovery of all containers Sep 10 13:17:18.416181 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:17:18.416150 2557 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-b9tk7" Sep 10 13:17:18.416934 ip-10-0-139-174 kubenswrapper[2557]: E0910 13:17:18.416906 2557 reflector.go:205] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver" Sep 10 13:17:18.417033 ip-10-0-139-174 kubenswrapper[2557]: E0910 13:17:18.416977 2557 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-139-174.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms" Sep 10 13:17:18.417710 ip-10-0-139-174 kubenswrapper[2557]: E0910 13:17:18.417690 2557 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service" Sep 10 13:17:18.417769 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:17:18.417756 2557 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-139-174.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Sep 10 13:17:18.417851 ip-10-0-139-174 kubenswrapper[2557]: E0910 13:17:18.417834 2557 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-139-174.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node" Sep 10 13:17:18.418176 ip-10-0-139-174 kubenswrapper[2557]: E0910 13:17:18.417132 2557 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-139-174.ec2.internal.18d3f81336dd191d default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-139-174.ec2.internal,UID:ip-10-0-139-174.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-139-174.ec2.internal,},FirstTimestamp:2026-09-10 13:17:18.404745501 +0000 UTC m=+0.361093980,LastTimestamp:2026-09-10 13:17:18.404745501 +0000 UTC m=+0.361093980,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-139-174.ec2.internal,}" Sep 10 13:17:18.425767 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:17:18.425741 2557 manager.go:354] Recovery completed Sep 10 13:17:18.428300 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:17:18.428279 2557 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-b9tk7" Sep 10 13:17:18.430875 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:17:18.430858 2557 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 10 13:17:18.432560 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:17:18.432539 2557 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-174.ec2.internal" event="NodeHasSufficientMemory" Sep 10 13:17:18.432631 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:17:18.432582 2557 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-174.ec2.internal" event="NodeHasNoDiskPressure" Sep 10 13:17:18.432631 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:17:18.432594 2557 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-174.ec2.internal" event="NodeHasSufficientPID" Sep 10 13:17:18.433128 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:17:18.433112 2557 cpu_manager.go:222] "Starting CPU manager" policy="none" Sep 10 13:17:18.433128 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:17:18.433127 2557 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s" Sep 10 13:17:18.433265 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:17:18.433145 2557 state_mem.go:36] "Initialized new in-memory state store" Sep 10 13:17:18.433265 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:17:18.433119 2557 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 10 13:17:18.433855 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:17:18.433841 2557 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-174.ec2.internal" event="NodeHasSufficientMemory" Sep 10 13:17:18.433904 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:17:18.433867 2557 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-174.ec2.internal" event="NodeHasNoDiskPressure" Sep 10 13:17:18.433904 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:17:18.433877 2557 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-174.ec2.internal" event="NodeHasSufficientPID" Sep 10 13:17:18.434398 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:17:18.434385 2557 policy_none.go:49] "None policy: Start" Sep 10 13:17:18.434455 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:17:18.434401 2557 memory_manager.go:187] "Starting memorymanager" policy="None" Sep 10 13:17:18.434455 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:17:18.434411 2557 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint" Sep 10 13:17:18.435390 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:17:18.435379 2557 policy_none.go:47] "Start" Sep 10 13:17:18.485534 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:17:18.471870 2557 manager.go:337] "Starting Device Plugin manager" Sep 10 13:17:18.485534 ip-10-0-139-174 kubenswrapper[2557]: E0910 13:17:18.471901 2557 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint" Sep 10 13:17:18.485534 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:17:18.471911 2557 server.go:87] "Starting device plugin registration server" Sep 10 13:17:18.485534 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:17:18.472236 2557 eviction_manager.go:189] "Eviction manager: starting control loop" Sep 10 13:17:18.485534 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:17:18.472252 2557 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s" Sep 10 13:17:18.485534 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:17:18.472364 2557 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry" Sep 10 13:17:18.485534 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:17:18.472507 2557 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts" Sep 10 13:17:18.485534 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:17:18.472517 2557 plugin_manager.go:118] "Starting Kubelet Plugin Manager" Sep 10 13:17:18.485534 ip-10-0-139-174 kubenswrapper[2557]: E0910 13:17:18.472880 2557 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\"" Sep 10 13:17:18.485534 ip-10-0-139-174 kubenswrapper[2557]: E0910 13:17:18.472925 2557 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-139-174.ec2.internal\" not found" Sep 10 13:17:18.562439 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:17:18.562417 2557 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4" Sep 10 13:17:18.563663 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:17:18.563641 2557 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6" Sep 10 13:17:18.563749 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:17:18.563704 2557 status_manager.go:244] "Starting to sync pod status with apiserver" Sep 10 13:17:18.563749 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:17:18.563729 2557 kubelet.go:2448] "Starting kubelet main sync loop" Sep 10 13:17:18.563832 ip-10-0-139-174 kubenswrapper[2557]: E0910 13:17:18.563773 2557 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful" Sep 10 13:17:18.566562 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:17:18.566547 2557 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160" Sep 10 13:17:18.572603 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:17:18.572582 2557 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 10 13:17:18.573375 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:17:18.573354 2557 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-174.ec2.internal" event="NodeHasSufficientMemory" Sep 10 13:17:18.573457 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:17:18.573385 2557 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-174.ec2.internal" event="NodeHasNoDiskPressure" Sep 10 13:17:18.573457 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:17:18.573397 2557 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-174.ec2.internal" event="NodeHasSufficientPID" Sep 10 13:17:18.573457 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:17:18.573416 2557 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-139-174.ec2.internal" Sep 10 13:17:18.581981 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:17:18.581921 2557 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-139-174.ec2.internal" Sep 10 13:17:18.581981 ip-10-0-139-174 kubenswrapper[2557]: E0910 13:17:18.581942 2557 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-139-174.ec2.internal\": node \"ip-10-0-139-174.ec2.internal\" not found" Sep 10 13:17:18.604238 ip-10-0-139-174 kubenswrapper[2557]: E0910 13:17:18.604220 2557 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-139-174.ec2.internal\" not found" Sep 10 13:17:18.664041 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:17:18.664022 2557 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-174.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-139-174.ec2.internal"] Sep 10 13:17:18.664126 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:17:18.664110 2557 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 10 13:17:18.664902 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:17:18.664880 2557 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-174.ec2.internal" event="NodeHasSufficientMemory" Sep 10 13:17:18.664902 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:17:18.664905 2557 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-174.ec2.internal" event="NodeHasNoDiskPressure" Sep 10 13:17:18.665033 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:17:18.664915 2557 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-174.ec2.internal" event="NodeHasSufficientPID" Sep 10 13:17:18.666255 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:17:18.666243 2557 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 10 13:17:18.666363 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:17:18.666342 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-174.ec2.internal" Sep 10 13:17:18.666414 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:17:18.666377 2557 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 10 13:17:18.666931 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:17:18.666917 2557 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-174.ec2.internal" event="NodeHasSufficientMemory" Sep 10 13:17:18.666992 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:17:18.666931 2557 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-174.ec2.internal" event="NodeHasSufficientMemory" Sep 10 13:17:18.666992 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:17:18.666943 2557 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-174.ec2.internal" event="NodeHasNoDiskPressure" Sep 10 13:17:18.666992 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:17:18.666953 2557 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-174.ec2.internal" event="NodeHasNoDiskPressure" Sep 10 13:17:18.666992 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:17:18.666964 2557 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-174.ec2.internal" event="NodeHasSufficientPID" Sep 10 13:17:18.667129 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:17:18.666955 2557 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-174.ec2.internal" event="NodeHasSufficientPID" Sep 10 13:17:18.668226 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:17:18.668211 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-139-174.ec2.internal" Sep 10 13:17:18.668280 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:17:18.668242 2557 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 10 13:17:18.668868 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:17:18.668851 2557 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-174.ec2.internal" event="NodeHasSufficientMemory" Sep 10 13:17:18.668937 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:17:18.668880 2557 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-174.ec2.internal" event="NodeHasNoDiskPressure" Sep 10 13:17:18.668937 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:17:18.668894 2557 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-174.ec2.internal" event="NodeHasSufficientPID" Sep 10 13:17:18.690625 ip-10-0-139-174 kubenswrapper[2557]: E0910 13:17:18.690606 2557 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-139-174.ec2.internal\" not found" node="ip-10-0-139-174.ec2.internal" Sep 10 13:17:18.694872 ip-10-0-139-174 kubenswrapper[2557]: E0910 13:17:18.694857 2557 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-139-174.ec2.internal\" not found" node="ip-10-0-139-174.ec2.internal" Sep 10 13:17:18.705030 ip-10-0-139-174 kubenswrapper[2557]: E0910 13:17:18.705012 2557 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-139-174.ec2.internal\" not found" Sep 10 13:17:18.716407 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:17:18.716386 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/329bcfe776c7597f50776992405be196-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-139-174.ec2.internal\" (UID: \"329bcfe776c7597f50776992405be196\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-174.ec2.internal" Sep 10 13:17:18.716496 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:17:18.716416 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/329bcfe776c7597f50776992405be196-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-139-174.ec2.internal\" (UID: \"329bcfe776c7597f50776992405be196\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-174.ec2.internal" Sep 10 13:17:18.716496 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:17:18.716442 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/b2d5d97b9a9b3ad7edee5cc4d2502cd5-config\") pod \"kube-apiserver-proxy-ip-10-0-139-174.ec2.internal\" (UID: \"b2d5d97b9a9b3ad7edee5cc4d2502cd5\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-139-174.ec2.internal" Sep 10 13:17:18.805823 ip-10-0-139-174 kubenswrapper[2557]: E0910 13:17:18.805805 2557 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-139-174.ec2.internal\" not found" Sep 10 13:17:18.817178 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:17:18.817155 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/329bcfe776c7597f50776992405be196-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-139-174.ec2.internal\" (UID: \"329bcfe776c7597f50776992405be196\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-174.ec2.internal" Sep 10 13:17:18.817252 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:17:18.817182 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/b2d5d97b9a9b3ad7edee5cc4d2502cd5-config\") pod \"kube-apiserver-proxy-ip-10-0-139-174.ec2.internal\" (UID: \"b2d5d97b9a9b3ad7edee5cc4d2502cd5\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-139-174.ec2.internal" Sep 10 13:17:18.817252 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:17:18.817196 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/329bcfe776c7597f50776992405be196-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-139-174.ec2.internal\" (UID: \"329bcfe776c7597f50776992405be196\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-174.ec2.internal" Sep 10 13:17:18.817252 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:17:18.817236 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/329bcfe776c7597f50776992405be196-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-139-174.ec2.internal\" (UID: \"329bcfe776c7597f50776992405be196\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-174.ec2.internal" Sep 10 13:17:18.817380 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:17:18.817248 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/329bcfe776c7597f50776992405be196-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-139-174.ec2.internal\" (UID: \"329bcfe776c7597f50776992405be196\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-174.ec2.internal" Sep 10 13:17:18.817380 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:17:18.817287 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/b2d5d97b9a9b3ad7edee5cc4d2502cd5-config\") pod \"kube-apiserver-proxy-ip-10-0-139-174.ec2.internal\" (UID: \"b2d5d97b9a9b3ad7edee5cc4d2502cd5\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-139-174.ec2.internal" Sep 10 13:17:18.905929 ip-10-0-139-174 kubenswrapper[2557]: E0910 13:17:18.905870 2557 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-139-174.ec2.internal\" not found" Sep 10 13:17:18.993370 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:17:18.993350 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-174.ec2.internal" Sep 10 13:17:18.998103 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:17:18.998091 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-139-174.ec2.internal" Sep 10 13:17:19.006867 ip-10-0-139-174 kubenswrapper[2557]: E0910 13:17:19.006851 2557 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-139-174.ec2.internal\" not found" Sep 10 13:17:19.107332 ip-10-0-139-174 kubenswrapper[2557]: E0910 13:17:19.107300 2557 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-139-174.ec2.internal\" not found" Sep 10 13:17:19.207894 ip-10-0-139-174 kubenswrapper[2557]: E0910 13:17:19.207843 2557 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-139-174.ec2.internal\" not found" Sep 10 13:17:19.253640 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:17:19.253616 2557 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160" Sep 10 13:17:19.308039 ip-10-0-139-174 kubenswrapper[2557]: E0910 13:17:19.308009 2557 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-139-174.ec2.internal\" not found" Sep 10 13:17:19.348484 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:17:19.348467 2557 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials" Sep 10 13:17:19.348843 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:17:19.348563 2557 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received" Sep 10 13:17:19.348843 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:17:19.348603 2557 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received" Sep 10 13:17:19.409019 ip-10-0-139-174 kubenswrapper[2557]: E0910 13:17:19.408992 2557 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-139-174.ec2.internal\" not found" Sep 10 13:17:19.430212 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:17:19.430177 2557 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-09-09 13:12:20 +0000 UTC" deadline="2028-05-12 00:32:05.34532302 +0000 UTC" Sep 10 13:17:19.430306 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:17:19.430218 2557 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="14627h14m45.915123064s" Sep 10 13:17:19.487334 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:17:19.487307 2557 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160" Sep 10 13:17:19.513660 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:17:19.513454 2557 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-174.ec2.internal" Sep 10 13:17:19.527015 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:17:19.526994 2557 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Sep 10 13:17:19.528176 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:17:19.528159 2557 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-139-174.ec2.internal" Sep 10 13:17:19.535168 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:17:19.535148 2557 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Sep 10 13:17:19.610985 ip-10-0-139-174 kubenswrapper[2557]: W0910 13:17:19.610951 2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod329bcfe776c7597f50776992405be196.slice/crio-ec9780f17457ec262978758a83f314a7d0743cffc967ebdf1df3e005579fd746 WatchSource:0}: Error finding container ec9780f17457ec262978758a83f314a7d0743cffc967ebdf1df3e005579fd746: Status 404 returned error can't find the container with id ec9780f17457ec262978758a83f314a7d0743cffc967ebdf1df3e005579fd746 Sep 10 13:17:19.617258 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:17:19.617241 2557 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Sep 10 13:17:19.666514 ip-10-0-139-174 kubenswrapper[2557]: W0910 13:17:19.666493 2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb2d5d97b9a9b3ad7edee5cc4d2502cd5.slice/crio-e84c492337ddd5531eda0c375c9a9f8aab3b14bf4775838d019b3c0728a05705 WatchSource:0}: Error finding container e84c492337ddd5531eda0c375c9a9f8aab3b14bf4775838d019b3c0728a05705: Status 404 returned error can't find the container with id e84c492337ddd5531eda0c375c9a9f8aab3b14bf4775838d019b3c0728a05705 Sep 10 13:17:19.681166 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:17:19.681148 2557 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160" Sep 10 13:17:22.041084 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:17:22.041055 2557 apiserver.go:52] "Watching apiserver" Sep 10 13:17:22.047467 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:17:22.047441 2557 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66" Sep 10 13:17:22.049673 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:17:22.049641 2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-multus/multus-additional-cni-plugins-sctq4","openshift-multus/network-metrics-daemon-7vvkj","kube-system/kube-apiserver-proxy-ip-10-0-139-174.ec2.internal","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-174.ec2.internal","openshift-multus/multus-65w2f","openshift-network-diagnostics/network-check-target-6ggtx","openshift-network-operator/iptables-alerter-c22v2","openshift-ovn-kubernetes/ovnkube-node-92n4q","kube-system/global-pull-secret-syncer-phs5r","kube-system/konnectivity-agent-kfhgq","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qnjfm","openshift-cluster-node-tuning-operator/tuned-g95cv","openshift-dns/node-resolver-6wf8v","openshift-image-registry/node-ca-zzkks"] Sep 10 13:17:22.051338 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:17:22.051310 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-kfhgq" Sep 10 13:17:22.053753 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:17:22.053728 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-nkkrx\"" Sep 10 13:17:22.053985 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:17:22.053967 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\"" Sep 10 13:17:22.054577 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:17:22.054558 2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\"" Sep 10 13:17:22.057535 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:17:22.057513 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-6wf8v" Sep 10 13:17:22.058117 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:17:22.058102 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-zzkks" Sep 10 13:17:22.058685 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:17:22.058669 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-92n4q" Sep 10 13:17:22.059689 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:17:22.059670 2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\"" Sep 10 13:17:22.059799 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:17:22.059730 2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\"" Sep 10 13:17:22.059799 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:17:22.059765 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-9hbcs\"" Sep 10 13:17:22.059942 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:17:22.059914 2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\"" Sep 10 13:17:22.059942 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:17:22.059920 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-65w2f" Sep 10 13:17:22.060322 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:17:22.060303 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-6ggtx" Sep 10 13:17:22.060414 ip-10-0-139-174 kubenswrapper[2557]: E0910 13:17:22.060368 2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-6ggtx" podUID="accbb486-a5c9-469d-beca-837d2bbd4eab" Sep 10 13:17:22.061022 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:17:22.061008 2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\"" Sep 10 13:17:22.061136 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:17:22.061010 2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\"" Sep 10 13:17:22.061203 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:17:22.061016 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-dvrz6\"" Sep 10 13:17:22.061257 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:17:22.061064 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\"" Sep 10 13:17:22.061800 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:17:22.061078 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-4gngz\"" Sep 10 13:17:22.061800 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:17:22.061100 2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\"" Sep 10 13:17:22.061800 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:17:22.061502 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-phs5r" Sep 10 13:17:22.061800 ip-10-0-139-174 kubenswrapper[2557]: E0910 13:17:22.061553 2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-phs5r" podUID="d873f3a0-ccac-4d59-a6f6-2c93cf5118de" Sep 10 13:17:22.062014 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:17:22.061819 2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\"" Sep 10 13:17:22.062014 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:17:22.061825 2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\"" Sep 10 13:17:22.062014 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:17:22.061829 2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\"" Sep 10 13:17:22.062510 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:17:22.062376 2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\"" Sep 10 13:17:22.062510 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:17:22.062408 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-857rb\"" Sep 10 13:17:22.062510 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:17:22.062506 2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\"" Sep 10 13:17:22.062657 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:17:22.062453 2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\"" Sep 10 13:17:22.063168 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:17:22.062870 2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\"" Sep 10 13:17:22.063168 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:17:22.063009 2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\"" Sep 10 13:17:22.063301 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:17:22.063249 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-c22v2" Sep 10 13:17:22.063602 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:17:22.063580 2557 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving" Sep 10 13:17:22.068950 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:17:22.067902 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-7vvkj" Sep 10 13:17:22.068950 ip-10-0-139-174 kubenswrapper[2557]: E0910 13:17:22.068024 2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-7vvkj" podUID="4016c772-d4cc-4de5-9272-f33e9da5b2b8" Sep 10 13:17:22.068950 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:17:22.068070 2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\"" Sep 10 13:17:22.068950 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:17:22.068304 2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\"" Sep 10 13:17:22.068950 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:17:22.068575 2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\"" Sep 10 13:17:22.069894 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:17:22.069007 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-d7tsw\"" Sep 10 13:17:22.070132 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:17:22.070118 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-sctq4" Sep 10 13:17:22.071937 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:17:22.071919 2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\"" Sep 10 13:17:22.072609 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:17:22.072223 2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\"" Sep 10 13:17:22.072609 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:17:22.072265 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-86gg7\"" Sep 10 13:17:22.072990 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:17:22.072970 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qnjfm" Sep 10 13:17:22.076615 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:17:22.073937 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/8fe2ceee-5166-41b0-a39c-5732ea7e336b-etc-kubernetes\") pod \"multus-65w2f\" (UID: \"8fe2ceee-5166-41b0-a39c-5732ea7e336b\") " pod="openshift-multus/multus-65w2f" Sep 10 13:17:22.076615 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:17:22.073969 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/a13f63dc-c50e-4f46-84fe-d9718d8b2308-agent-certs\") pod \"konnectivity-agent-kfhgq\" (UID: \"a13f63dc-c50e-4f46-84fe-d9718d8b2308\") " pod="kube-system/konnectivity-agent-kfhgq" Sep 10 13:17:22.076615 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:17:22.073993 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/720acda6-0b15-49ac-ab4e-60de70c4324a-etc-openvswitch\") pod \"ovnkube-node-92n4q\" (UID: \"720acda6-0b15-49ac-ab4e-60de70c4324a\") " pod="openshift-ovn-kubernetes/ovnkube-node-92n4q" Sep 10 13:17:22.076615 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:17:22.074015 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/720acda6-0b15-49ac-ab4e-60de70c4324a-host-cni-bin\") pod \"ovnkube-node-92n4q\" (UID: \"720acda6-0b15-49ac-ab4e-60de70c4324a\") " pod="openshift-ovn-kubernetes/ovnkube-node-92n4q" Sep 10 13:17:22.076615 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:17:22.074038 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/a13f63dc-c50e-4f46-84fe-d9718d8b2308-konnectivity-ca\") pod \"konnectivity-agent-kfhgq\" (UID: \"a13f63dc-c50e-4f46-84fe-d9718d8b2308\") " pod="kube-system/konnectivity-agent-kfhgq" Sep 10 13:17:22.076615 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:17:22.074061 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/720acda6-0b15-49ac-ab4e-60de70c4324a-host-slash\") pod \"ovnkube-node-92n4q\" (UID: \"720acda6-0b15-49ac-ab4e-60de70c4324a\") " pod="openshift-ovn-kubernetes/ovnkube-node-92n4q" Sep 10 13:17:22.076615 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:17:22.074086 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/720acda6-0b15-49ac-ab4e-60de70c4324a-run-systemd\") pod \"ovnkube-node-92n4q\" (UID: \"720acda6-0b15-49ac-ab4e-60de70c4324a\") " pod="openshift-ovn-kubernetes/ovnkube-node-92n4q" Sep 10 13:17:22.076615 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:17:22.074126 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/720acda6-0b15-49ac-ab4e-60de70c4324a-var-lib-openvswitch\") pod \"ovnkube-node-92n4q\" (UID: \"720acda6-0b15-49ac-ab4e-60de70c4324a\") " pod="openshift-ovn-kubernetes/ovnkube-node-92n4q" Sep 10 13:17:22.076615 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:17:22.074159 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/720acda6-0b15-49ac-ab4e-60de70c4324a-run-ovn\") pod \"ovnkube-node-92n4q\" (UID: \"720acda6-0b15-49ac-ab4e-60de70c4324a\") " pod="openshift-ovn-kubernetes/ovnkube-node-92n4q" Sep 10 13:17:22.076615 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:17:22.074192 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/720acda6-0b15-49ac-ab4e-60de70c4324a-ovnkube-script-lib\") pod \"ovnkube-node-92n4q\" (UID: \"720acda6-0b15-49ac-ab4e-60de70c4324a\") " pod="openshift-ovn-kubernetes/ovnkube-node-92n4q" Sep 10 13:17:22.076615 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:17:22.074223 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dww57\" (UniqueName: \"kubernetes.io/projected/720acda6-0b15-49ac-ab4e-60de70c4324a-kube-api-access-dww57\") pod \"ovnkube-node-92n4q\" (UID: \"720acda6-0b15-49ac-ab4e-60de70c4324a\") " pod="openshift-ovn-kubernetes/ovnkube-node-92n4q" Sep 10 13:17:22.076615 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:17:22.074251 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/8fe2ceee-5166-41b0-a39c-5732ea7e336b-multus-cni-dir\") pod \"multus-65w2f\" (UID: \"8fe2ceee-5166-41b0-a39c-5732ea7e336b\") " pod="openshift-multus/multus-65w2f" Sep 10 13:17:22.076615 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:17:22.074292 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/8fe2ceee-5166-41b0-a39c-5732ea7e336b-multus-socket-dir-parent\") pod \"multus-65w2f\" (UID: \"8fe2ceee-5166-41b0-a39c-5732ea7e336b\") " pod="openshift-multus/multus-65w2f" Sep 10 13:17:22.076615 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:17:22.074326 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/8fe2ceee-5166-41b0-a39c-5732ea7e336b-host-run-k8s-cni-cncf-io\") pod \"multus-65w2f\" (UID: \"8fe2ceee-5166-41b0-a39c-5732ea7e336b\") " pod="openshift-multus/multus-65w2f" Sep 10 13:17:22.076615 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:17:22.074353 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/8fe2ceee-5166-41b0-a39c-5732ea7e336b-host-run-netns\") pod \"multus-65w2f\" (UID: \"8fe2ceee-5166-41b0-a39c-5732ea7e336b\") " pod="openshift-multus/multus-65w2f" Sep 10 13:17:22.076615 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:17:22.074380 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/8fe2ceee-5166-41b0-a39c-5732ea7e336b-host-var-lib-cni-bin\") pod \"multus-65w2f\" (UID: \"8fe2ceee-5166-41b0-a39c-5732ea7e336b\") " pod="openshift-multus/multus-65w2f" Sep 10 13:17:22.076615 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:17:22.074408 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/8fe2ceee-5166-41b0-a39c-5732ea7e336b-multus-daemon-config\") pod \"multus-65w2f\" (UID: \"8fe2ceee-5166-41b0-a39c-5732ea7e336b\") " pod="openshift-multus/multus-65w2f" Sep 10 13:17:22.077445 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:17:22.074386 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-g95cv" Sep 10 13:17:22.077445 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:17:22.074449 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/720acda6-0b15-49ac-ab4e-60de70c4324a-log-socket\") pod \"ovnkube-node-92n4q\" (UID: \"720acda6-0b15-49ac-ab4e-60de70c4324a\") " pod="openshift-ovn-kubernetes/ovnkube-node-92n4q" Sep 10 13:17:22.077445 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:17:22.074481 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/720acda6-0b15-49ac-ab4e-60de70c4324a-env-overrides\") pod \"ovnkube-node-92n4q\" (UID: \"720acda6-0b15-49ac-ab4e-60de70c4324a\") " pod="openshift-ovn-kubernetes/ovnkube-node-92n4q" Sep 10 13:17:22.077445 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:17:22.074509 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/9e7f9da8-1931-42da-b5c4-67542de73659-host\") pod \"node-ca-zzkks\" (UID: \"9e7f9da8-1931-42da-b5c4-67542de73659\") " pod="openshift-image-registry/node-ca-zzkks" Sep 10 13:17:22.077445 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:17:22.074531 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/9e7f9da8-1931-42da-b5c4-67542de73659-serviceca\") pod \"node-ca-zzkks\" (UID: \"9e7f9da8-1931-42da-b5c4-67542de73659\") " pod="openshift-image-registry/node-ca-zzkks" Sep 10 13:17:22.077445 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:17:22.074550 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9vgjt\" (UniqueName: \"kubernetes.io/projected/9e7f9da8-1931-42da-b5c4-67542de73659-kube-api-access-9vgjt\") pod \"node-ca-zzkks\" (UID: \"9e7f9da8-1931-42da-b5c4-67542de73659\") " pod="openshift-image-registry/node-ca-zzkks" Sep 10 13:17:22.077445 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:17:22.074566 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/8fe2ceee-5166-41b0-a39c-5732ea7e336b-host-var-lib-kubelet\") pod \"multus-65w2f\" (UID: \"8fe2ceee-5166-41b0-a39c-5732ea7e336b\") " pod="openshift-multus/multus-65w2f" Sep 10 13:17:22.077445 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:17:22.074587 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r6pw5\" (UniqueName: \"kubernetes.io/projected/accbb486-a5c9-469d-beca-837d2bbd4eab-kube-api-access-r6pw5\") pod \"network-check-target-6ggtx\" (UID: \"accbb486-a5c9-469d-beca-837d2bbd4eab\") " pod="openshift-network-diagnostics/network-check-target-6ggtx" Sep 10 13:17:22.077445 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:17:22.074601 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4016c772-d4cc-4de5-9272-f33e9da5b2b8-metrics-certs\") pod \"network-metrics-daemon-7vvkj\" (UID: \"4016c772-d4cc-4de5-9272-f33e9da5b2b8\") " pod="openshift-multus/network-metrics-daemon-7vvkj" Sep 10 13:17:22.077445 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:17:22.074614 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/720acda6-0b15-49ac-ab4e-60de70c4324a-run-openvswitch\") pod \"ovnkube-node-92n4q\" (UID: \"720acda6-0b15-49ac-ab4e-60de70c4324a\") " pod="openshift-ovn-kubernetes/ovnkube-node-92n4q" Sep 10 13:17:22.077445 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:17:22.074632 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/720acda6-0b15-49ac-ab4e-60de70c4324a-host-cni-netd\") pod \"ovnkube-node-92n4q\" (UID: \"720acda6-0b15-49ac-ab4e-60de70c4324a\") " pod="openshift-ovn-kubernetes/ovnkube-node-92n4q" Sep 10 13:17:22.077445 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:17:22.074679 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/8fe2ceee-5166-41b0-a39c-5732ea7e336b-system-cni-dir\") pod \"multus-65w2f\" (UID: \"8fe2ceee-5166-41b0-a39c-5732ea7e336b\") " pod="openshift-multus/multus-65w2f" Sep 10 13:17:22.077445 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:17:22.074699 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/8fe2ceee-5166-41b0-a39c-5732ea7e336b-cnibin\") pod \"multus-65w2f\" (UID: \"8fe2ceee-5166-41b0-a39c-5732ea7e336b\") " pod="openshift-multus/multus-65w2f" Sep 10 13:17:22.077445 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:17:22.074730 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/8fe2ceee-5166-41b0-a39c-5732ea7e336b-hostroot\") pod \"multus-65w2f\" (UID: \"8fe2ceee-5166-41b0-a39c-5732ea7e336b\") " pod="openshift-multus/multus-65w2f" Sep 10 13:17:22.077445 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:17:22.074757 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/d873f3a0-ccac-4d59-a6f6-2c93cf5118de-kubelet-config\") pod \"global-pull-secret-syncer-phs5r\" (UID: \"d873f3a0-ccac-4d59-a6f6-2c93cf5118de\") " pod="kube-system/global-pull-secret-syncer-phs5r" Sep 10 13:17:22.077445 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:17:22.074773 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f2904741-875c-40e6-a97e-e56ae66d5077-tmp-dir\") pod \"node-resolver-6wf8v\" (UID: \"f2904741-875c-40e6-a97e-e56ae66d5077\") " pod="openshift-dns/node-resolver-6wf8v" Sep 10 13:17:22.077445 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:17:22.074827 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/720acda6-0b15-49ac-ab4e-60de70c4324a-host-run-netns\") pod \"ovnkube-node-92n4q\" (UID: \"720acda6-0b15-49ac-ab4e-60de70c4324a\") " pod="openshift-ovn-kubernetes/ovnkube-node-92n4q" Sep 10 13:17:22.078284 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:17:22.074889 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/720acda6-0b15-49ac-ab4e-60de70c4324a-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-92n4q\" (UID: \"720acda6-0b15-49ac-ab4e-60de70c4324a\") " pod="openshift-ovn-kubernetes/ovnkube-node-92n4q" Sep 10 13:17:22.078284 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:17:22.074917 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/720acda6-0b15-49ac-ab4e-60de70c4324a-ovn-node-metrics-cert\") pod \"ovnkube-node-92n4q\" (UID: \"720acda6-0b15-49ac-ab4e-60de70c4324a\") " pod="openshift-ovn-kubernetes/ovnkube-node-92n4q" Sep 10 13:17:22.078284 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:17:22.074947 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/8fe2ceee-5166-41b0-a39c-5732ea7e336b-host-var-lib-cni-multus\") pod \"multus-65w2f\" (UID: \"8fe2ceee-5166-41b0-a39c-5732ea7e336b\") " pod="openshift-multus/multus-65w2f" Sep 10 13:17:22.078284 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:17:22.074974 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\"" Sep 10 13:17:22.078284 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:17:22.075024 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-jcw6z\"" Sep 10 13:17:22.078284 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:17:22.074975 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bzm29\" (UniqueName: \"kubernetes.io/projected/f2904741-875c-40e6-a97e-e56ae66d5077-kube-api-access-bzm29\") pod \"node-resolver-6wf8v\" (UID: \"f2904741-875c-40e6-a97e-e56ae66d5077\") " pod="openshift-dns/node-resolver-6wf8v" Sep 10 13:17:22.078284 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:17:22.075080 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/720acda6-0b15-49ac-ab4e-60de70c4324a-host-run-ovn-kubernetes\") pod \"ovnkube-node-92n4q\" (UID: \"720acda6-0b15-49ac-ab4e-60de70c4324a\") " pod="openshift-ovn-kubernetes/ovnkube-node-92n4q" Sep 10 13:17:22.078284 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:17:22.075107 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hst8g\" (UniqueName: \"kubernetes.io/projected/981d9727-9493-4d26-ac17-5fae22a92e0d-kube-api-access-hst8g\") pod \"iptables-alerter-c22v2\" (UID: \"981d9727-9493-4d26-ac17-5fae22a92e0d\") " pod="openshift-network-operator/iptables-alerter-c22v2" Sep 10 13:17:22.078284 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:17:22.075134 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/8fe2ceee-5166-41b0-a39c-5732ea7e336b-multus-conf-dir\") pod \"multus-65w2f\" (UID: \"8fe2ceee-5166-41b0-a39c-5732ea7e336b\") " pod="openshift-multus/multus-65w2f" Sep 10 13:17:22.078284 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:17:22.075159 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/8fe2ceee-5166-41b0-a39c-5732ea7e336b-host-run-multus-certs\") pod \"multus-65w2f\" (UID: \"8fe2ceee-5166-41b0-a39c-5732ea7e336b\") " pod="openshift-multus/multus-65w2f" Sep 10 13:17:22.078284 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:17:22.075165 2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\"" Sep 10 13:17:22.078284 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:17:22.075183 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4n6lh\" (UniqueName: \"kubernetes.io/projected/8fe2ceee-5166-41b0-a39c-5732ea7e336b-kube-api-access-4n6lh\") pod \"multus-65w2f\" (UID: \"8fe2ceee-5166-41b0-a39c-5732ea7e336b\") " pod="openshift-multus/multus-65w2f" Sep 10 13:17:22.078284 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:17:22.075208 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/d873f3a0-ccac-4d59-a6f6-2c93cf5118de-dbus\") pod \"global-pull-secret-syncer-phs5r\" (UID: \"d873f3a0-ccac-4d59-a6f6-2c93cf5118de\") " pod="kube-system/global-pull-secret-syncer-phs5r" Sep 10 13:17:22.078284 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:17:22.075233 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/d873f3a0-ccac-4d59-a6f6-2c93cf5118de-original-pull-secret\") pod \"global-pull-secret-syncer-phs5r\" (UID: \"d873f3a0-ccac-4d59-a6f6-2c93cf5118de\") " pod="kube-system/global-pull-secret-syncer-phs5r" Sep 10 13:17:22.078284 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:17:22.075257 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/f2904741-875c-40e6-a97e-e56ae66d5077-hosts-file\") pod \"node-resolver-6wf8v\" (UID: \"f2904741-875c-40e6-a97e-e56ae66d5077\") " pod="openshift-dns/node-resolver-6wf8v" Sep 10 13:17:22.078284 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:17:22.075274 2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\"" Sep 10 13:17:22.078284 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:17:22.075282 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/720acda6-0b15-49ac-ab4e-60de70c4324a-host-kubelet\") pod \"ovnkube-node-92n4q\" (UID: \"720acda6-0b15-49ac-ab4e-60de70c4324a\") " pod="openshift-ovn-kubernetes/ovnkube-node-92n4q" Sep 10 13:17:22.078284 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:17:22.075310 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/8fe2ceee-5166-41b0-a39c-5732ea7e336b-cni-binary-copy\") pod \"multus-65w2f\" (UID: \"8fe2ceee-5166-41b0-a39c-5732ea7e336b\") " pod="openshift-multus/multus-65w2f" Sep 10 13:17:22.079107 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:17:22.075347 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ljj8w\" (UniqueName: \"kubernetes.io/projected/4016c772-d4cc-4de5-9272-f33e9da5b2b8-kube-api-access-ljj8w\") pod \"network-metrics-daemon-7vvkj\" (UID: \"4016c772-d4cc-4de5-9272-f33e9da5b2b8\") " pod="openshift-multus/network-metrics-daemon-7vvkj" Sep 10 13:17:22.079107 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:17:22.075380 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/720acda6-0b15-49ac-ab4e-60de70c4324a-systemd-units\") pod \"ovnkube-node-92n4q\" (UID: \"720acda6-0b15-49ac-ab4e-60de70c4324a\") " pod="openshift-ovn-kubernetes/ovnkube-node-92n4q" Sep 10 13:17:22.079107 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:17:22.075432 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/720acda6-0b15-49ac-ab4e-60de70c4324a-node-log\") pod \"ovnkube-node-92n4q\" (UID: \"720acda6-0b15-49ac-ab4e-60de70c4324a\") " pod="openshift-ovn-kubernetes/ovnkube-node-92n4q" Sep 10 13:17:22.079107 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:17:22.075474 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/720acda6-0b15-49ac-ab4e-60de70c4324a-ovnkube-config\") pod \"ovnkube-node-92n4q\" (UID: \"720acda6-0b15-49ac-ab4e-60de70c4324a\") " pod="openshift-ovn-kubernetes/ovnkube-node-92n4q" Sep 10 13:17:22.079107 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:17:22.075500 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/981d9727-9493-4d26-ac17-5fae22a92e0d-iptables-alerter-script\") pod \"iptables-alerter-c22v2\" (UID: \"981d9727-9493-4d26-ac17-5fae22a92e0d\") " pod="openshift-network-operator/iptables-alerter-c22v2" Sep 10 13:17:22.079107 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:17:22.075524 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/981d9727-9493-4d26-ac17-5fae22a92e0d-host-slash\") pod \"iptables-alerter-c22v2\" (UID: \"981d9727-9493-4d26-ac17-5fae22a92e0d\") " pod="openshift-network-operator/iptables-alerter-c22v2" Sep 10 13:17:22.079107 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:17:22.076229 2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\"" Sep 10 13:17:22.079107 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:17:22.076330 2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\"" Sep 10 13:17:22.079107 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:17:22.076745 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-nw6bg\"" Sep 10 13:17:22.079107 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:17:22.077458 2557 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Sep 10 13:17:22.104207 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:17:22.104188 2557 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-rfjbt" Sep 10 13:17:22.112118 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:17:22.112098 2557 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-rfjbt" Sep 10 13:17:22.164128 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:17:22.164107 2557 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world" Sep 10 13:17:22.175766 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:17:22.175738 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/720acda6-0b15-49ac-ab4e-60de70c4324a-run-ovn\") pod \"ovnkube-node-92n4q\" (UID: \"720acda6-0b15-49ac-ab4e-60de70c4324a\") " pod="openshift-ovn-kubernetes/ovnkube-node-92n4q" Sep 10 13:17:22.175876 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:17:22.175778 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/8fe2ceee-5166-41b0-a39c-5732ea7e336b-multus-cni-dir\") pod \"multus-65w2f\" (UID: \"8fe2ceee-5166-41b0-a39c-5732ea7e336b\") " pod="openshift-multus/multus-65w2f" Sep 10 13:17:22.175876 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:17:22.175849 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/720acda6-0b15-49ac-ab4e-60de70c4324a-run-ovn\") pod \"ovnkube-node-92n4q\" (UID: \"720acda6-0b15-49ac-ab4e-60de70c4324a\") " pod="openshift-ovn-kubernetes/ovnkube-node-92n4q" Sep 10 13:17:22.175983 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:17:22.175891 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/8fe2ceee-5166-41b0-a39c-5732ea7e336b-multus-socket-dir-parent\") pod \"multus-65w2f\" (UID: \"8fe2ceee-5166-41b0-a39c-5732ea7e336b\") " pod="openshift-multus/multus-65w2f" Sep 10 13:17:22.175983 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:17:22.175923 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/8fe2ceee-5166-41b0-a39c-5732ea7e336b-multus-cni-dir\") pod \"multus-65w2f\" (UID: \"8fe2ceee-5166-41b0-a39c-5732ea7e336b\") " pod="openshift-multus/multus-65w2f" Sep 10 13:17:22.176093 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:17:22.175984 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/8fe2ceee-5166-41b0-a39c-5732ea7e336b-host-run-netns\") pod \"multus-65w2f\" (UID: \"8fe2ceee-5166-41b0-a39c-5732ea7e336b\") " pod="openshift-multus/multus-65w2f" Sep 10 13:17:22.176144 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:17:22.176129 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/8fe2ceee-5166-41b0-a39c-5732ea7e336b-multus-daemon-config\") pod \"multus-65w2f\" (UID: \"8fe2ceee-5166-41b0-a39c-5732ea7e336b\") " pod="openshift-multus/multus-65w2f" Sep 10 13:17:22.176194 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:17:22.176068 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/8fe2ceee-5166-41b0-a39c-5732ea7e336b-host-run-netns\") pod \"multus-65w2f\" (UID: \"8fe2ceee-5166-41b0-a39c-5732ea7e336b\") " pod="openshift-multus/multus-65w2f" Sep 10 13:17:22.176194 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:17:22.176159 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/720acda6-0b15-49ac-ab4e-60de70c4324a-log-socket\") pod \"ovnkube-node-92n4q\" (UID: \"720acda6-0b15-49ac-ab4e-60de70c4324a\") " pod="openshift-ovn-kubernetes/ovnkube-node-92n4q" Sep 10 13:17:22.176194 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:17:22.176183 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/720acda6-0b15-49ac-ab4e-60de70c4324a-env-overrides\") pod \"ovnkube-node-92n4q\" (UID: \"720acda6-0b15-49ac-ab4e-60de70c4324a\") " pod="openshift-ovn-kubernetes/ovnkube-node-92n4q" Sep 10 13:17:22.176438 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:17:22.176210 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/2873d0b1-1e1d-4c45-a663-5b25c9308427-cni-binary-copy\") pod \"multus-additional-cni-plugins-sctq4\" (UID: \"2873d0b1-1e1d-4c45-a663-5b25c9308427\") " pod="openshift-multus/multus-additional-cni-plugins-sctq4" Sep 10 13:17:22.176438 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:17:22.176235 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/9e7f9da8-1931-42da-b5c4-67542de73659-host\") pod \"node-ca-zzkks\" (UID: \"9e7f9da8-1931-42da-b5c4-67542de73659\") " pod="openshift-image-registry/node-ca-zzkks" Sep 10 13:17:22.176438 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:17:22.176259 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9vgjt\" (UniqueName: \"kubernetes.io/projected/9e7f9da8-1931-42da-b5c4-67542de73659-kube-api-access-9vgjt\") pod \"node-ca-zzkks\" (UID: \"9e7f9da8-1931-42da-b5c4-67542de73659\") " pod="openshift-image-registry/node-ca-zzkks" Sep 10 13:17:22.176438 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:17:22.176284 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r6pw5\" (UniqueName: \"kubernetes.io/projected/accbb486-a5c9-469d-beca-837d2bbd4eab-kube-api-access-r6pw5\") pod \"network-check-target-6ggtx\" (UID: \"accbb486-a5c9-469d-beca-837d2bbd4eab\") " pod="openshift-network-diagnostics/network-check-target-6ggtx" Sep 10 13:17:22.176438 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:17:22.176307 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4016c772-d4cc-4de5-9272-f33e9da5b2b8-metrics-certs\") pod \"network-metrics-daemon-7vvkj\" (UID: \"4016c772-d4cc-4de5-9272-f33e9da5b2b8\") " pod="openshift-multus/network-metrics-daemon-7vvkj" Sep 10 13:17:22.176438 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:17:22.176330 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/720acda6-0b15-49ac-ab4e-60de70c4324a-host-cni-netd\") pod \"ovnkube-node-92n4q\" (UID: \"720acda6-0b15-49ac-ab4e-60de70c4324a\") " pod="openshift-ovn-kubernetes/ovnkube-node-92n4q" Sep 10 13:17:22.176438 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:17:22.176353 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/720acda6-0b15-49ac-ab4e-60de70c4324a-ovnkube-script-lib\") pod \"ovnkube-node-92n4q\" (UID: \"720acda6-0b15-49ac-ab4e-60de70c4324a\") " pod="openshift-ovn-kubernetes/ovnkube-node-92n4q" Sep 10 13:17:22.176438 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:17:22.176376 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/2873d0b1-1e1d-4c45-a663-5b25c9308427-tuning-conf-dir\") pod \"multus-additional-cni-plugins-sctq4\" (UID: \"2873d0b1-1e1d-4c45-a663-5b25c9308427\") " pod="openshift-multus/multus-additional-cni-plugins-sctq4" Sep 10 13:17:22.176438 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:17:22.176400 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/2873d0b1-1e1d-4c45-a663-5b25c9308427-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-sctq4\" (UID: \"2873d0b1-1e1d-4c45-a663-5b25c9308427\") " pod="openshift-multus/multus-additional-cni-plugins-sctq4" Sep 10 13:17:22.176851 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:17:22.176457 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/8fe2ceee-5166-41b0-a39c-5732ea7e336b-cnibin\") pod \"multus-65w2f\" (UID: \"8fe2ceee-5166-41b0-a39c-5732ea7e336b\") " pod="openshift-multus/multus-65w2f" Sep 10 13:17:22.176851 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:17:22.176480 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f2904741-875c-40e6-a97e-e56ae66d5077-tmp-dir\") pod \"node-resolver-6wf8v\" (UID: \"f2904741-875c-40e6-a97e-e56ae66d5077\") " pod="openshift-dns/node-resolver-6wf8v" Sep 10 13:17:22.176851 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:17:22.176504 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tc5wf\" (UniqueName: \"kubernetes.io/projected/2873d0b1-1e1d-4c45-a663-5b25c9308427-kube-api-access-tc5wf\") pod \"multus-additional-cni-plugins-sctq4\" (UID: \"2873d0b1-1e1d-4c45-a663-5b25c9308427\") " pod="openshift-multus/multus-additional-cni-plugins-sctq4" Sep 10 13:17:22.176851 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:17:22.176529 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/bf875c9d-61a1-4b2f-8089-9b80ecb1c689-etc-kubernetes\") pod \"tuned-g95cv\" (UID: \"bf875c9d-61a1-4b2f-8089-9b80ecb1c689\") " pod="openshift-cluster-node-tuning-operator/tuned-g95cv" Sep 10 13:17:22.176851 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:17:22.176556 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/720acda6-0b15-49ac-ab4e-60de70c4324a-host-run-ovn-kubernetes\") pod \"ovnkube-node-92n4q\" (UID: \"720acda6-0b15-49ac-ab4e-60de70c4324a\") " pod="openshift-ovn-kubernetes/ovnkube-node-92n4q" Sep 10 13:17:22.176851 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:17:22.176659 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/bf875c9d-61a1-4b2f-8089-9b80ecb1c689-var-lib-kubelet\") pod \"tuned-g95cv\" (UID: \"bf875c9d-61a1-4b2f-8089-9b80ecb1c689\") " pod="openshift-cluster-node-tuning-operator/tuned-g95cv" Sep 10 13:17:22.176851 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:17:22.176233 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/720acda6-0b15-49ac-ab4e-60de70c4324a-log-socket\") pod \"ovnkube-node-92n4q\" (UID: \"720acda6-0b15-49ac-ab4e-60de70c4324a\") " pod="openshift-ovn-kubernetes/ovnkube-node-92n4q" Sep 10 13:17:22.176851 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:17:22.176682 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/bf875c9d-61a1-4b2f-8089-9b80ecb1c689-host\") pod \"tuned-g95cv\" (UID: \"bf875c9d-61a1-4b2f-8089-9b80ecb1c689\") " pod="openshift-cluster-node-tuning-operator/tuned-g95cv" Sep 10 13:17:22.176851 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:17:22.176132 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/8fe2ceee-5166-41b0-a39c-5732ea7e336b-multus-socket-dir-parent\") pod \"multus-65w2f\" (UID: \"8fe2ceee-5166-41b0-a39c-5732ea7e336b\") " pod="openshift-multus/multus-65w2f" Sep 10 13:17:22.176851 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:17:22.176707 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/8fe2ceee-5166-41b0-a39c-5732ea7e336b-host-run-multus-certs\") pod \"multus-65w2f\" (UID: \"8fe2ceee-5166-41b0-a39c-5732ea7e336b\") " pod="openshift-multus/multus-65w2f" Sep 10 13:17:22.176851 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:17:22.176760 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/8fe2ceee-5166-41b0-a39c-5732ea7e336b-host-run-multus-certs\") pod \"multus-65w2f\" (UID: \"8fe2ceee-5166-41b0-a39c-5732ea7e336b\") " pod="openshift-multus/multus-65w2f" Sep 10 13:17:22.176851 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:17:22.176821 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/8fe2ceee-5166-41b0-a39c-5732ea7e336b-cnibin\") pod \"multus-65w2f\" (UID: \"8fe2ceee-5166-41b0-a39c-5732ea7e336b\") " pod="openshift-multus/multus-65w2f" Sep 10 13:17:22.177412 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:17:22.176908 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/8fe2ceee-5166-41b0-a39c-5732ea7e336b-multus-daemon-config\") pod \"multus-65w2f\" (UID: \"8fe2ceee-5166-41b0-a39c-5732ea7e336b\") " pod="openshift-multus/multus-65w2f" Sep 10 13:17:22.177412 ip-10-0-139-174 kubenswrapper[2557]: E0910 13:17:22.176973 2557 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Sep 10 13:17:22.177412 ip-10-0-139-174 kubenswrapper[2557]: E0910 13:17:22.177044 2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4016c772-d4cc-4de5-9272-f33e9da5b2b8-metrics-certs podName:4016c772-d4cc-4de5-9272-f33e9da5b2b8 nodeName:}" failed. No retries permitted until 2026-09-10 13:17:22.677024703 +0000 UTC m=+2.984019114 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/4016c772-d4cc-4de5-9272-f33e9da5b2b8-metrics-certs") pod "network-metrics-daemon-7vvkj" (UID: "4016c772-d4cc-4de5-9272-f33e9da5b2b8") : object "openshift-multus"/"metrics-daemon-secret" not registered Sep 10 13:17:22.177412 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:17:22.177098 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f2904741-875c-40e6-a97e-e56ae66d5077-tmp-dir\") pod \"node-resolver-6wf8v\" (UID: \"f2904741-875c-40e6-a97e-e56ae66d5077\") " pod="openshift-dns/node-resolver-6wf8v" Sep 10 13:17:22.177412 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:17:22.177186 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/720acda6-0b15-49ac-ab4e-60de70c4324a-host-run-ovn-kubernetes\") pod \"ovnkube-node-92n4q\" (UID: \"720acda6-0b15-49ac-ab4e-60de70c4324a\") " pod="openshift-ovn-kubernetes/ovnkube-node-92n4q" Sep 10 13:17:22.177412 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:17:22.177218 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/9e7f9da8-1931-42da-b5c4-67542de73659-host\") pod \"node-ca-zzkks\" (UID: \"9e7f9da8-1931-42da-b5c4-67542de73659\") " pod="openshift-image-registry/node-ca-zzkks" Sep 10 13:17:22.177412 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:17:22.177232 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4n6lh\" (UniqueName: \"kubernetes.io/projected/8fe2ceee-5166-41b0-a39c-5732ea7e336b-kube-api-access-4n6lh\") pod \"multus-65w2f\" (UID: \"8fe2ceee-5166-41b0-a39c-5732ea7e336b\") " pod="openshift-multus/multus-65w2f" Sep 10 13:17:22.177412 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:17:22.177250 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/720acda6-0b15-49ac-ab4e-60de70c4324a-host-cni-netd\") pod \"ovnkube-node-92n4q\" (UID: \"720acda6-0b15-49ac-ab4e-60de70c4324a\") " pod="openshift-ovn-kubernetes/ovnkube-node-92n4q" Sep 10 13:17:22.177412 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:17:22.177272 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/d873f3a0-ccac-4d59-a6f6-2c93cf5118de-dbus\") pod \"global-pull-secret-syncer-phs5r\" (UID: \"d873f3a0-ccac-4d59-a6f6-2c93cf5118de\") " pod="kube-system/global-pull-secret-syncer-phs5r" Sep 10 13:17:22.177412 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:17:22.177297 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/d873f3a0-ccac-4d59-a6f6-2c93cf5118de-original-pull-secret\") pod \"global-pull-secret-syncer-phs5r\" (UID: \"d873f3a0-ccac-4d59-a6f6-2c93cf5118de\") " pod="kube-system/global-pull-secret-syncer-phs5r" Sep 10 13:17:22.177412 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:17:22.177319 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/f2904741-875c-40e6-a97e-e56ae66d5077-hosts-file\") pod \"node-resolver-6wf8v\" (UID: \"f2904741-875c-40e6-a97e-e56ae66d5077\") " pod="openshift-dns/node-resolver-6wf8v" Sep 10 13:17:22.177412 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:17:22.177340 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/720acda6-0b15-49ac-ab4e-60de70c4324a-env-overrides\") pod \"ovnkube-node-92n4q\" (UID: \"720acda6-0b15-49ac-ab4e-60de70c4324a\") " pod="openshift-ovn-kubernetes/ovnkube-node-92n4q" Sep 10 13:17:22.177412 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:17:22.177344 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/720acda6-0b15-49ac-ab4e-60de70c4324a-host-kubelet\") pod \"ovnkube-node-92n4q\" (UID: \"720acda6-0b15-49ac-ab4e-60de70c4324a\") " pod="openshift-ovn-kubernetes/ovnkube-node-92n4q" Sep 10 13:17:22.177412 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:17:22.177379 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/720acda6-0b15-49ac-ab4e-60de70c4324a-host-kubelet\") pod \"ovnkube-node-92n4q\" (UID: \"720acda6-0b15-49ac-ab4e-60de70c4324a\") " pod="openshift-ovn-kubernetes/ovnkube-node-92n4q" Sep 10 13:17:22.177412 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:17:22.177396 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/720acda6-0b15-49ac-ab4e-60de70c4324a-node-log\") pod \"ovnkube-node-92n4q\" (UID: \"720acda6-0b15-49ac-ab4e-60de70c4324a\") " pod="openshift-ovn-kubernetes/ovnkube-node-92n4q" Sep 10 13:17:22.177412 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:17:22.177434 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/720acda6-0b15-49ac-ab4e-60de70c4324a-ovnkube-config\") pod \"ovnkube-node-92n4q\" (UID: \"720acda6-0b15-49ac-ab4e-60de70c4324a\") " pod="openshift-ovn-kubernetes/ovnkube-node-92n4q" Sep 10 13:17:22.178158 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:17:22.177466 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/f2904741-875c-40e6-a97e-e56ae66d5077-hosts-file\") pod \"node-resolver-6wf8v\" (UID: \"f2904741-875c-40e6-a97e-e56ae66d5077\") " pod="openshift-dns/node-resolver-6wf8v" Sep 10 13:17:22.178158 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:17:22.177500 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/8fe2ceee-5166-41b0-a39c-5732ea7e336b-cni-binary-copy\") pod \"multus-65w2f\" (UID: \"8fe2ceee-5166-41b0-a39c-5732ea7e336b\") " pod="openshift-multus/multus-65w2f" Sep 10 13:17:22.178158 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:17:22.177515 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/720acda6-0b15-49ac-ab4e-60de70c4324a-node-log\") pod \"ovnkube-node-92n4q\" (UID: \"720acda6-0b15-49ac-ab4e-60de70c4324a\") " pod="openshift-ovn-kubernetes/ovnkube-node-92n4q" Sep 10 13:17:22.178158 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:17:22.177545 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ljj8w\" (UniqueName: \"kubernetes.io/projected/4016c772-d4cc-4de5-9272-f33e9da5b2b8-kube-api-access-ljj8w\") pod \"network-metrics-daemon-7vvkj\" (UID: \"4016c772-d4cc-4de5-9272-f33e9da5b2b8\") " pod="openshift-multus/network-metrics-daemon-7vvkj" Sep 10 13:17:22.178158 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:17:22.177551 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/d873f3a0-ccac-4d59-a6f6-2c93cf5118de-dbus\") pod \"global-pull-secret-syncer-phs5r\" (UID: \"d873f3a0-ccac-4d59-a6f6-2c93cf5118de\") " pod="kube-system/global-pull-secret-syncer-phs5r" Sep 10 13:17:22.178158 ip-10-0-139-174 kubenswrapper[2557]: E0910 13:17:22.177515 2557 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Sep 10 13:17:22.178158 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:17:22.177575 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/720acda6-0b15-49ac-ab4e-60de70c4324a-systemd-units\") pod \"ovnkube-node-92n4q\" (UID: \"720acda6-0b15-49ac-ab4e-60de70c4324a\") " pod="openshift-ovn-kubernetes/ovnkube-node-92n4q" Sep 10 13:17:22.178158 ip-10-0-139-174 kubenswrapper[2557]: E0910 13:17:22.177605 2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d873f3a0-ccac-4d59-a6f6-2c93cf5118de-original-pull-secret podName:d873f3a0-ccac-4d59-a6f6-2c93cf5118de nodeName:}" failed. No retries permitted until 2026-09-10 13:17:22.677595303 +0000 UTC m=+2.984589696 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/d873f3a0-ccac-4d59-a6f6-2c93cf5118de-original-pull-secret") pod "global-pull-secret-syncer-phs5r" (UID: "d873f3a0-ccac-4d59-a6f6-2c93cf5118de") : object "kube-system"/"original-pull-secret" not registered Sep 10 13:17:22.178158 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:17:22.177618 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/eb907ce0-50ee-4062-9c00-5ede30885af1-kubelet-dir\") pod \"aws-ebs-csi-driver-node-qnjfm\" (UID: \"eb907ce0-50ee-4062-9c00-5ede30885af1\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qnjfm" Sep 10 13:17:22.178158 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:17:22.177635 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/720acda6-0b15-49ac-ab4e-60de70c4324a-systemd-units\") pod \"ovnkube-node-92n4q\" (UID: \"720acda6-0b15-49ac-ab4e-60de70c4324a\") " pod="openshift-ovn-kubernetes/ovnkube-node-92n4q" Sep 10 13:17:22.178158 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:17:22.177668 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/eb907ce0-50ee-4062-9c00-5ede30885af1-socket-dir\") pod \"aws-ebs-csi-driver-node-qnjfm\" (UID: \"eb907ce0-50ee-4062-9c00-5ede30885af1\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qnjfm" Sep 10 13:17:22.178158 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:17:22.177697 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/720acda6-0b15-49ac-ab4e-60de70c4324a-host-cni-bin\") pod \"ovnkube-node-92n4q\" (UID: \"720acda6-0b15-49ac-ab4e-60de70c4324a\") " pod="openshift-ovn-kubernetes/ovnkube-node-92n4q" Sep 10 13:17:22.178158 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:17:22.177725 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/eb907ce0-50ee-4062-9c00-5ede30885af1-registration-dir\") pod \"aws-ebs-csi-driver-node-qnjfm\" (UID: \"eb907ce0-50ee-4062-9c00-5ede30885af1\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qnjfm" Sep 10 13:17:22.178158 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:17:22.177750 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/720acda6-0b15-49ac-ab4e-60de70c4324a-host-cni-bin\") pod \"ovnkube-node-92n4q\" (UID: \"720acda6-0b15-49ac-ab4e-60de70c4324a\") " pod="openshift-ovn-kubernetes/ovnkube-node-92n4q" Sep 10 13:17:22.178158 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:17:22.177755 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/a13f63dc-c50e-4f46-84fe-d9718d8b2308-konnectivity-ca\") pod \"konnectivity-agent-kfhgq\" (UID: \"a13f63dc-c50e-4f46-84fe-d9718d8b2308\") " pod="kube-system/konnectivity-agent-kfhgq" Sep 10 13:17:22.178158 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:17:22.177778 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/720acda6-0b15-49ac-ab4e-60de70c4324a-host-slash\") pod \"ovnkube-node-92n4q\" (UID: \"720acda6-0b15-49ac-ab4e-60de70c4324a\") " pod="openshift-ovn-kubernetes/ovnkube-node-92n4q" Sep 10 13:17:22.178158 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:17:22.177803 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/720acda6-0b15-49ac-ab4e-60de70c4324a-var-lib-openvswitch\") pod \"ovnkube-node-92n4q\" (UID: \"720acda6-0b15-49ac-ab4e-60de70c4324a\") " pod="openshift-ovn-kubernetes/ovnkube-node-92n4q" Sep 10 13:17:22.178974 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:17:22.177831 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dww57\" (UniqueName: \"kubernetes.io/projected/720acda6-0b15-49ac-ab4e-60de70c4324a-kube-api-access-dww57\") pod \"ovnkube-node-92n4q\" (UID: \"720acda6-0b15-49ac-ab4e-60de70c4324a\") " pod="openshift-ovn-kubernetes/ovnkube-node-92n4q" Sep 10 13:17:22.178974 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:17:22.177856 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/bf875c9d-61a1-4b2f-8089-9b80ecb1c689-tmp\") pod \"tuned-g95cv\" (UID: \"bf875c9d-61a1-4b2f-8089-9b80ecb1c689\") " pod="openshift-cluster-node-tuning-operator/tuned-g95cv" Sep 10 13:17:22.178974 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:17:22.177882 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vcwwn\" (UniqueName: \"kubernetes.io/projected/bf875c9d-61a1-4b2f-8089-9b80ecb1c689-kube-api-access-vcwwn\") pod \"tuned-g95cv\" (UID: \"bf875c9d-61a1-4b2f-8089-9b80ecb1c689\") " pod="openshift-cluster-node-tuning-operator/tuned-g95cv" Sep 10 13:17:22.178974 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:17:22.177906 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/720acda6-0b15-49ac-ab4e-60de70c4324a-host-slash\") pod \"ovnkube-node-92n4q\" (UID: \"720acda6-0b15-49ac-ab4e-60de70c4324a\") " pod="openshift-ovn-kubernetes/ovnkube-node-92n4q" Sep 10 13:17:22.178974 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:17:22.177924 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/8fe2ceee-5166-41b0-a39c-5732ea7e336b-host-run-k8s-cni-cncf-io\") pod \"multus-65w2f\" (UID: \"8fe2ceee-5166-41b0-a39c-5732ea7e336b\") " pod="openshift-multus/multus-65w2f" Sep 10 13:17:22.178974 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:17:22.177939 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/720acda6-0b15-49ac-ab4e-60de70c4324a-var-lib-openvswitch\") pod \"ovnkube-node-92n4q\" (UID: \"720acda6-0b15-49ac-ab4e-60de70c4324a\") " pod="openshift-ovn-kubernetes/ovnkube-node-92n4q" Sep 10 13:17:22.178974 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:17:22.177963 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/8fe2ceee-5166-41b0-a39c-5732ea7e336b-host-var-lib-cni-bin\") pod \"multus-65w2f\" (UID: \"8fe2ceee-5166-41b0-a39c-5732ea7e336b\") " pod="openshift-multus/multus-65w2f" Sep 10 13:17:22.178974 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:17:22.177983 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/8fe2ceee-5166-41b0-a39c-5732ea7e336b-host-run-k8s-cni-cncf-io\") pod \"multus-65w2f\" (UID: \"8fe2ceee-5166-41b0-a39c-5732ea7e336b\") " pod="openshift-multus/multus-65w2f" Sep 10 13:17:22.178974 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:17:22.177989 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hst8g\" (UniqueName: \"kubernetes.io/projected/981d9727-9493-4d26-ac17-5fae22a92e0d-kube-api-access-hst8g\") pod \"iptables-alerter-c22v2\" (UID: \"981d9727-9493-4d26-ac17-5fae22a92e0d\") " pod="openshift-network-operator/iptables-alerter-c22v2" Sep 10 13:17:22.178974 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:17:22.178017 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/bf875c9d-61a1-4b2f-8089-9b80ecb1c689-run\") pod \"tuned-g95cv\" (UID: \"bf875c9d-61a1-4b2f-8089-9b80ecb1c689\") " pod="openshift-cluster-node-tuning-operator/tuned-g95cv" Sep 10 13:17:22.178974 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:17:22.178031 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/720acda6-0b15-49ac-ab4e-60de70c4324a-ovnkube-script-lib\") pod \"ovnkube-node-92n4q\" (UID: \"720acda6-0b15-49ac-ab4e-60de70c4324a\") " pod="openshift-ovn-kubernetes/ovnkube-node-92n4q" Sep 10 13:17:22.178974 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:17:22.178040 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/bf875c9d-61a1-4b2f-8089-9b80ecb1c689-etc-tuned\") pod \"tuned-g95cv\" (UID: \"bf875c9d-61a1-4b2f-8089-9b80ecb1c689\") " pod="openshift-cluster-node-tuning-operator/tuned-g95cv" Sep 10 13:17:22.178974 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:17:22.178033 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/8fe2ceee-5166-41b0-a39c-5732ea7e336b-host-var-lib-cni-bin\") pod \"multus-65w2f\" (UID: \"8fe2ceee-5166-41b0-a39c-5732ea7e336b\") " pod="openshift-multus/multus-65w2f" Sep 10 13:17:22.178974 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:17:22.178064 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/9e7f9da8-1931-42da-b5c4-67542de73659-serviceca\") pod \"node-ca-zzkks\" (UID: \"9e7f9da8-1931-42da-b5c4-67542de73659\") " pod="openshift-image-registry/node-ca-zzkks" Sep 10 13:17:22.178974 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:17:22.178096 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/8fe2ceee-5166-41b0-a39c-5732ea7e336b-host-var-lib-kubelet\") pod \"multus-65w2f\" (UID: \"8fe2ceee-5166-41b0-a39c-5732ea7e336b\") " pod="openshift-multus/multus-65w2f" Sep 10 13:17:22.178974 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:17:22.178120 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/720acda6-0b15-49ac-ab4e-60de70c4324a-run-openvswitch\") pod \"ovnkube-node-92n4q\" (UID: \"720acda6-0b15-49ac-ab4e-60de70c4324a\") " pod="openshift-ovn-kubernetes/ovnkube-node-92n4q" Sep 10 13:17:22.178974 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:17:22.178123 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/720acda6-0b15-49ac-ab4e-60de70c4324a-ovnkube-config\") pod \"ovnkube-node-92n4q\" (UID: \"720acda6-0b15-49ac-ab4e-60de70c4324a\") " pod="openshift-ovn-kubernetes/ovnkube-node-92n4q" Sep 10 13:17:22.179788 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:17:22.178122 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/8fe2ceee-5166-41b0-a39c-5732ea7e336b-cni-binary-copy\") pod \"multus-65w2f\" (UID: \"8fe2ceee-5166-41b0-a39c-5732ea7e336b\") " pod="openshift-multus/multus-65w2f" Sep 10 13:17:22.179788 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:17:22.178143 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/720acda6-0b15-49ac-ab4e-60de70c4324a-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-92n4q\" (UID: \"720acda6-0b15-49ac-ab4e-60de70c4324a\") " pod="openshift-ovn-kubernetes/ovnkube-node-92n4q" Sep 10 13:17:22.179788 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:17:22.178167 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/720acda6-0b15-49ac-ab4e-60de70c4324a-ovn-node-metrics-cert\") pod \"ovnkube-node-92n4q\" (UID: \"720acda6-0b15-49ac-ab4e-60de70c4324a\") " pod="openshift-ovn-kubernetes/ovnkube-node-92n4q" Sep 10 13:17:22.179788 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:17:22.178180 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/8fe2ceee-5166-41b0-a39c-5732ea7e336b-host-var-lib-kubelet\") pod \"multus-65w2f\" (UID: \"8fe2ceee-5166-41b0-a39c-5732ea7e336b\") " pod="openshift-multus/multus-65w2f" Sep 10 13:17:22.179788 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:17:22.178223 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/720acda6-0b15-49ac-ab4e-60de70c4324a-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-92n4q\" (UID: \"720acda6-0b15-49ac-ab4e-60de70c4324a\") " pod="openshift-ovn-kubernetes/ovnkube-node-92n4q" Sep 10 13:17:22.179788 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:17:22.178192 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/eb907ce0-50ee-4062-9c00-5ede30885af1-device-dir\") pod \"aws-ebs-csi-driver-node-qnjfm\" (UID: \"eb907ce0-50ee-4062-9c00-5ede30885af1\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qnjfm" Sep 10 13:17:22.179788 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:17:22.178249 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/720acda6-0b15-49ac-ab4e-60de70c4324a-run-openvswitch\") pod \"ovnkube-node-92n4q\" (UID: \"720acda6-0b15-49ac-ab4e-60de70c4324a\") " pod="openshift-ovn-kubernetes/ovnkube-node-92n4q" Sep 10 13:17:22.179788 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:17:22.178265 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7kdmf\" (UniqueName: \"kubernetes.io/projected/eb907ce0-50ee-4062-9c00-5ede30885af1-kube-api-access-7kdmf\") pod \"aws-ebs-csi-driver-node-qnjfm\" (UID: \"eb907ce0-50ee-4062-9c00-5ede30885af1\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qnjfm" Sep 10 13:17:22.179788 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:17:22.178294 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/8fe2ceee-5166-41b0-a39c-5732ea7e336b-system-cni-dir\") pod \"multus-65w2f\" (UID: \"8fe2ceee-5166-41b0-a39c-5732ea7e336b\") " pod="openshift-multus/multus-65w2f" Sep 10 13:17:22.179788 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:17:22.178324 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/8fe2ceee-5166-41b0-a39c-5732ea7e336b-hostroot\") pod \"multus-65w2f\" (UID: \"8fe2ceee-5166-41b0-a39c-5732ea7e336b\") " pod="openshift-multus/multus-65w2f" Sep 10 13:17:22.179788 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:17:22.178354 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/d873f3a0-ccac-4d59-a6f6-2c93cf5118de-kubelet-config\") pod \"global-pull-secret-syncer-phs5r\" (UID: \"d873f3a0-ccac-4d59-a6f6-2c93cf5118de\") " pod="kube-system/global-pull-secret-syncer-phs5r" Sep 10 13:17:22.179788 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:17:22.178374 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/720acda6-0b15-49ac-ab4e-60de70c4324a-host-run-netns\") pod \"ovnkube-node-92n4q\" (UID: \"720acda6-0b15-49ac-ab4e-60de70c4324a\") " pod="openshift-ovn-kubernetes/ovnkube-node-92n4q" Sep 10 13:17:22.179788 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:17:22.178406 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/8fe2ceee-5166-41b0-a39c-5732ea7e336b-system-cni-dir\") pod \"multus-65w2f\" (UID: \"8fe2ceee-5166-41b0-a39c-5732ea7e336b\") " pod="openshift-multus/multus-65w2f" Sep 10 13:17:22.179788 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:17:22.178437 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/d873f3a0-ccac-4d59-a6f6-2c93cf5118de-kubelet-config\") pod \"global-pull-secret-syncer-phs5r\" (UID: \"d873f3a0-ccac-4d59-a6f6-2c93cf5118de\") " pod="kube-system/global-pull-secret-syncer-phs5r" Sep 10 13:17:22.179788 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:17:22.178458 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/720acda6-0b15-49ac-ab4e-60de70c4324a-host-run-netns\") pod \"ovnkube-node-92n4q\" (UID: \"720acda6-0b15-49ac-ab4e-60de70c4324a\") " pod="openshift-ovn-kubernetes/ovnkube-node-92n4q" Sep 10 13:17:22.179788 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:17:22.178472 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/8fe2ceee-5166-41b0-a39c-5732ea7e336b-hostroot\") pod \"multus-65w2f\" (UID: \"8fe2ceee-5166-41b0-a39c-5732ea7e336b\") " pod="openshift-multus/multus-65w2f" Sep 10 13:17:22.179788 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:17:22.178472 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/a13f63dc-c50e-4f46-84fe-d9718d8b2308-konnectivity-ca\") pod \"konnectivity-agent-kfhgq\" (UID: \"a13f63dc-c50e-4f46-84fe-d9718d8b2308\") " pod="kube-system/konnectivity-agent-kfhgq" Sep 10 13:17:22.180585 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:17:22.178490 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/2873d0b1-1e1d-4c45-a663-5b25c9308427-cnibin\") pod \"multus-additional-cni-plugins-sctq4\" (UID: \"2873d0b1-1e1d-4c45-a663-5b25c9308427\") " pod="openshift-multus/multus-additional-cni-plugins-sctq4" Sep 10 13:17:22.180585 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:17:22.178519 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/2873d0b1-1e1d-4c45-a663-5b25c9308427-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-sctq4\" (UID: \"2873d0b1-1e1d-4c45-a663-5b25c9308427\") " pod="openshift-multus/multus-additional-cni-plugins-sctq4" Sep 10 13:17:22.180585 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:17:22.178542 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/bf875c9d-61a1-4b2f-8089-9b80ecb1c689-etc-modprobe-d\") pod \"tuned-g95cv\" (UID: \"bf875c9d-61a1-4b2f-8089-9b80ecb1c689\") " pod="openshift-cluster-node-tuning-operator/tuned-g95cv" Sep 10 13:17:22.180585 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:17:22.178549 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/9e7f9da8-1931-42da-b5c4-67542de73659-serviceca\") pod \"node-ca-zzkks\" (UID: \"9e7f9da8-1931-42da-b5c4-67542de73659\") " pod="openshift-image-registry/node-ca-zzkks" Sep 10 13:17:22.180585 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:17:22.178589 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/8fe2ceee-5166-41b0-a39c-5732ea7e336b-host-var-lib-cni-multus\") pod \"multus-65w2f\" (UID: \"8fe2ceee-5166-41b0-a39c-5732ea7e336b\") " pod="openshift-multus/multus-65w2f" Sep 10 13:17:22.180585 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:17:22.178613 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bzm29\" (UniqueName: \"kubernetes.io/projected/f2904741-875c-40e6-a97e-e56ae66d5077-kube-api-access-bzm29\") pod \"node-resolver-6wf8v\" (UID: \"f2904741-875c-40e6-a97e-e56ae66d5077\") " pod="openshift-dns/node-resolver-6wf8v" Sep 10 13:17:22.180585 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:17:22.178635 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/bf875c9d-61a1-4b2f-8089-9b80ecb1c689-etc-sysctl-conf\") pod \"tuned-g95cv\" (UID: \"bf875c9d-61a1-4b2f-8089-9b80ecb1c689\") " pod="openshift-cluster-node-tuning-operator/tuned-g95cv" Sep 10 13:17:22.180585 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:17:22.178660 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/8fe2ceee-5166-41b0-a39c-5732ea7e336b-multus-conf-dir\") pod \"multus-65w2f\" (UID: \"8fe2ceee-5166-41b0-a39c-5732ea7e336b\") " pod="openshift-multus/multus-65w2f" Sep 10 13:17:22.180585 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:17:22.178687 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/981d9727-9493-4d26-ac17-5fae22a92e0d-host-slash\") pod \"iptables-alerter-c22v2\" (UID: \"981d9727-9493-4d26-ac17-5fae22a92e0d\") " pod="openshift-network-operator/iptables-alerter-c22v2" Sep 10 13:17:22.180585 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:17:22.178704 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/8fe2ceee-5166-41b0-a39c-5732ea7e336b-multus-conf-dir\") pod \"multus-65w2f\" (UID: \"8fe2ceee-5166-41b0-a39c-5732ea7e336b\") " pod="openshift-multus/multus-65w2f" Sep 10 13:17:22.180585 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:17:22.178710 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/bf875c9d-61a1-4b2f-8089-9b80ecb1c689-etc-systemd\") pod \"tuned-g95cv\" (UID: \"bf875c9d-61a1-4b2f-8089-9b80ecb1c689\") " pod="openshift-cluster-node-tuning-operator/tuned-g95cv" Sep 10 13:17:22.180585 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:17:22.178707 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/8fe2ceee-5166-41b0-a39c-5732ea7e336b-host-var-lib-cni-multus\") pod \"multus-65w2f\" (UID: \"8fe2ceee-5166-41b0-a39c-5732ea7e336b\") " pod="openshift-multus/multus-65w2f" Sep 10 13:17:22.180585 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:17:22.178734 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/eb907ce0-50ee-4062-9c00-5ede30885af1-sys-fs\") pod \"aws-ebs-csi-driver-node-qnjfm\" (UID: \"eb907ce0-50ee-4062-9c00-5ede30885af1\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qnjfm" Sep 10 13:17:22.180585 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:17:22.178742 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/981d9727-9493-4d26-ac17-5fae22a92e0d-host-slash\") pod \"iptables-alerter-c22v2\" (UID: \"981d9727-9493-4d26-ac17-5fae22a92e0d\") " pod="openshift-network-operator/iptables-alerter-c22v2" Sep 10 13:17:22.180585 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:17:22.178774 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/981d9727-9493-4d26-ac17-5fae22a92e0d-iptables-alerter-script\") pod \"iptables-alerter-c22v2\" (UID: \"981d9727-9493-4d26-ac17-5fae22a92e0d\") " pod="openshift-network-operator/iptables-alerter-c22v2" Sep 10 13:17:22.180585 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:17:22.178812 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/2873d0b1-1e1d-4c45-a663-5b25c9308427-system-cni-dir\") pod \"multus-additional-cni-plugins-sctq4\" (UID: \"2873d0b1-1e1d-4c45-a663-5b25c9308427\") " pod="openshift-multus/multus-additional-cni-plugins-sctq4" Sep 10 13:17:22.180585 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:17:22.178836 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/bf875c9d-61a1-4b2f-8089-9b80ecb1c689-etc-sysconfig\") pod \"tuned-g95cv\" (UID: \"bf875c9d-61a1-4b2f-8089-9b80ecb1c689\") " pod="openshift-cluster-node-tuning-operator/tuned-g95cv" Sep 10 13:17:22.181353 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:17:22.178850 2557 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory" Sep 10 13:17:22.181353 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:17:22.178917 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/bf875c9d-61a1-4b2f-8089-9b80ecb1c689-etc-sysctl-d\") pod \"tuned-g95cv\" (UID: \"bf875c9d-61a1-4b2f-8089-9b80ecb1c689\") " pod="openshift-cluster-node-tuning-operator/tuned-g95cv" Sep 10 13:17:22.181353 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:17:22.178952 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/bf875c9d-61a1-4b2f-8089-9b80ecb1c689-sys\") pod \"tuned-g95cv\" (UID: \"bf875c9d-61a1-4b2f-8089-9b80ecb1c689\") " pod="openshift-cluster-node-tuning-operator/tuned-g95cv" Sep 10 13:17:22.181353 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:17:22.178982 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/8fe2ceee-5166-41b0-a39c-5732ea7e336b-etc-kubernetes\") pod \"multus-65w2f\" (UID: \"8fe2ceee-5166-41b0-a39c-5732ea7e336b\") " pod="openshift-multus/multus-65w2f" Sep 10 13:17:22.181353 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:17:22.179006 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/a13f63dc-c50e-4f46-84fe-d9718d8b2308-agent-certs\") pod \"konnectivity-agent-kfhgq\" (UID: \"a13f63dc-c50e-4f46-84fe-d9718d8b2308\") " pod="kube-system/konnectivity-agent-kfhgq" Sep 10 13:17:22.181353 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:17:22.179048 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/720acda6-0b15-49ac-ab4e-60de70c4324a-etc-openvswitch\") pod \"ovnkube-node-92n4q\" (UID: \"720acda6-0b15-49ac-ab4e-60de70c4324a\") " pod="openshift-ovn-kubernetes/ovnkube-node-92n4q" Sep 10 13:17:22.181353 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:17:22.179075 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/bf875c9d-61a1-4b2f-8089-9b80ecb1c689-lib-modules\") pod \"tuned-g95cv\" (UID: \"bf875c9d-61a1-4b2f-8089-9b80ecb1c689\") " pod="openshift-cluster-node-tuning-operator/tuned-g95cv" Sep 10 13:17:22.181353 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:17:22.179086 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/8fe2ceee-5166-41b0-a39c-5732ea7e336b-etc-kubernetes\") pod \"multus-65w2f\" (UID: \"8fe2ceee-5166-41b0-a39c-5732ea7e336b\") " pod="openshift-multus/multus-65w2f" Sep 10 13:17:22.181353 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:17:22.179099 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/eb907ce0-50ee-4062-9c00-5ede30885af1-etc-selinux\") pod \"aws-ebs-csi-driver-node-qnjfm\" (UID: \"eb907ce0-50ee-4062-9c00-5ede30885af1\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qnjfm" Sep 10 13:17:22.181353 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:17:22.179125 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/720acda6-0b15-49ac-ab4e-60de70c4324a-run-systemd\") pod \"ovnkube-node-92n4q\" (UID: \"720acda6-0b15-49ac-ab4e-60de70c4324a\") " pod="openshift-ovn-kubernetes/ovnkube-node-92n4q" Sep 10 13:17:22.181353 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:17:22.179187 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/720acda6-0b15-49ac-ab4e-60de70c4324a-etc-openvswitch\") pod \"ovnkube-node-92n4q\" (UID: \"720acda6-0b15-49ac-ab4e-60de70c4324a\") " pod="openshift-ovn-kubernetes/ovnkube-node-92n4q" Sep 10 13:17:22.181353 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:17:22.179229 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/720acda6-0b15-49ac-ab4e-60de70c4324a-run-systemd\") pod \"ovnkube-node-92n4q\" (UID: \"720acda6-0b15-49ac-ab4e-60de70c4324a\") " pod="openshift-ovn-kubernetes/ovnkube-node-92n4q" Sep 10 13:17:22.181353 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:17:22.179304 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/981d9727-9493-4d26-ac17-5fae22a92e0d-iptables-alerter-script\") pod \"iptables-alerter-c22v2\" (UID: \"981d9727-9493-4d26-ac17-5fae22a92e0d\") " pod="openshift-network-operator/iptables-alerter-c22v2" Sep 10 13:17:22.184283 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:17:22.184237 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/a13f63dc-c50e-4f46-84fe-d9718d8b2308-agent-certs\") pod \"konnectivity-agent-kfhgq\" (UID: \"a13f63dc-c50e-4f46-84fe-d9718d8b2308\") " pod="kube-system/konnectivity-agent-kfhgq" Sep 10 13:17:22.184395 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:17:22.184298 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/720acda6-0b15-49ac-ab4e-60de70c4324a-ovn-node-metrics-cert\") pod \"ovnkube-node-92n4q\" (UID: \"720acda6-0b15-49ac-ab4e-60de70c4324a\") " pod="openshift-ovn-kubernetes/ovnkube-node-92n4q" Sep 10 13:17:22.185554 ip-10-0-139-174 kubenswrapper[2557]: E0910 13:17:22.185533 2557 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Sep 10 13:17:22.185621 ip-10-0-139-174 kubenswrapper[2557]: E0910 13:17:22.185560 2557 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Sep 10 13:17:22.185621 ip-10-0-139-174 kubenswrapper[2557]: E0910 13:17:22.185572 2557 projected.go:196] Error preparing data for projected volume kube-api-access-r6pw5 for pod openshift-network-diagnostics/network-check-target-6ggtx: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 10 13:17:22.185621 ip-10-0-139-174 kubenswrapper[2557]: E0910 13:17:22.185619 2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/accbb486-a5c9-469d-beca-837d2bbd4eab-kube-api-access-r6pw5 podName:accbb486-a5c9-469d-beca-837d2bbd4eab nodeName:}" failed. No retries permitted until 2026-09-10 13:17:22.685604262 +0000 UTC m=+2.992598667 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-r6pw5" (UniqueName: "kubernetes.io/projected/accbb486-a5c9-469d-beca-837d2bbd4eab-kube-api-access-r6pw5") pod "network-check-target-6ggtx" (UID: "accbb486-a5c9-469d-beca-837d2bbd4eab") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 10 13:17:22.188249 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:17:22.188214 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4n6lh\" (UniqueName: \"kubernetes.io/projected/8fe2ceee-5166-41b0-a39c-5732ea7e336b-kube-api-access-4n6lh\") pod \"multus-65w2f\" (UID: \"8fe2ceee-5166-41b0-a39c-5732ea7e336b\") " pod="openshift-multus/multus-65w2f" Sep 10 13:17:22.189995 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:17:22.189975 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9vgjt\" (UniqueName: \"kubernetes.io/projected/9e7f9da8-1931-42da-b5c4-67542de73659-kube-api-access-9vgjt\") pod \"node-ca-zzkks\" (UID: \"9e7f9da8-1931-42da-b5c4-67542de73659\") " pod="openshift-image-registry/node-ca-zzkks" Sep 10 13:17:22.194405 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:17:22.194361 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ljj8w\" (UniqueName: \"kubernetes.io/projected/4016c772-d4cc-4de5-9272-f33e9da5b2b8-kube-api-access-ljj8w\") pod \"network-metrics-daemon-7vvkj\" (UID: \"4016c772-d4cc-4de5-9272-f33e9da5b2b8\") " pod="openshift-multus/network-metrics-daemon-7vvkj" Sep 10 13:17:22.196075 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:17:22.196049 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hst8g\" (UniqueName: \"kubernetes.io/projected/981d9727-9493-4d26-ac17-5fae22a92e0d-kube-api-access-hst8g\") pod \"iptables-alerter-c22v2\" (UID: \"981d9727-9493-4d26-ac17-5fae22a92e0d\") " pod="openshift-network-operator/iptables-alerter-c22v2" Sep 10 13:17:22.196806 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:17:22.196216 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dww57\" (UniqueName: \"kubernetes.io/projected/720acda6-0b15-49ac-ab4e-60de70c4324a-kube-api-access-dww57\") pod \"ovnkube-node-92n4q\" (UID: \"720acda6-0b15-49ac-ab4e-60de70c4324a\") " pod="openshift-ovn-kubernetes/ovnkube-node-92n4q" Sep 10 13:17:22.196897 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:17:22.196858 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bzm29\" (UniqueName: \"kubernetes.io/projected/f2904741-875c-40e6-a97e-e56ae66d5077-kube-api-access-bzm29\") pod \"node-resolver-6wf8v\" (UID: \"f2904741-875c-40e6-a97e-e56ae66d5077\") " pod="openshift-dns/node-resolver-6wf8v" Sep 10 13:17:22.220834 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:17:22.220780 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-139-174.ec2.internal" event={"ID":"b2d5d97b9a9b3ad7edee5cc4d2502cd5","Type":"ContainerStarted","Data":"e84c492337ddd5531eda0c375c9a9f8aab3b14bf4775838d019b3c0728a05705"} Sep 10 13:17:22.222478 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:17:22.222452 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-174.ec2.internal" event={"ID":"329bcfe776c7597f50776992405be196","Type":"ContainerStarted","Data":"ec9780f17457ec262978758a83f314a7d0743cffc967ebdf1df3e005579fd746"} Sep 10 13:17:22.279648 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:17:22.279618 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tc5wf\" (UniqueName: \"kubernetes.io/projected/2873d0b1-1e1d-4c45-a663-5b25c9308427-kube-api-access-tc5wf\") pod \"multus-additional-cni-plugins-sctq4\" (UID: \"2873d0b1-1e1d-4c45-a663-5b25c9308427\") " pod="openshift-multus/multus-additional-cni-plugins-sctq4" Sep 10 13:17:22.279801 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:17:22.279658 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/bf875c9d-61a1-4b2f-8089-9b80ecb1c689-etc-kubernetes\") pod \"tuned-g95cv\" (UID: \"bf875c9d-61a1-4b2f-8089-9b80ecb1c689\") " pod="openshift-cluster-node-tuning-operator/tuned-g95cv" Sep 10 13:17:22.279801 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:17:22.279682 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/bf875c9d-61a1-4b2f-8089-9b80ecb1c689-var-lib-kubelet\") pod \"tuned-g95cv\" (UID: \"bf875c9d-61a1-4b2f-8089-9b80ecb1c689\") " pod="openshift-cluster-node-tuning-operator/tuned-g95cv" Sep 10 13:17:22.279801 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:17:22.279704 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/bf875c9d-61a1-4b2f-8089-9b80ecb1c689-host\") pod \"tuned-g95cv\" (UID: \"bf875c9d-61a1-4b2f-8089-9b80ecb1c689\") " pod="openshift-cluster-node-tuning-operator/tuned-g95cv" Sep 10 13:17:22.279801 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:17:22.279753 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/eb907ce0-50ee-4062-9c00-5ede30885af1-kubelet-dir\") pod \"aws-ebs-csi-driver-node-qnjfm\" (UID: \"eb907ce0-50ee-4062-9c00-5ede30885af1\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qnjfm" Sep 10 13:17:22.279801 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:17:22.279778 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/eb907ce0-50ee-4062-9c00-5ede30885af1-socket-dir\") pod \"aws-ebs-csi-driver-node-qnjfm\" (UID: \"eb907ce0-50ee-4062-9c00-5ede30885af1\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qnjfm" Sep 10 13:17:22.279801 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:17:22.279800 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/eb907ce0-50ee-4062-9c00-5ede30885af1-registration-dir\") pod \"aws-ebs-csi-driver-node-qnjfm\" (UID: \"eb907ce0-50ee-4062-9c00-5ede30885af1\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qnjfm" Sep 10 13:17:22.280090 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:17:22.279825 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/bf875c9d-61a1-4b2f-8089-9b80ecb1c689-tmp\") pod \"tuned-g95cv\" (UID: \"bf875c9d-61a1-4b2f-8089-9b80ecb1c689\") " pod="openshift-cluster-node-tuning-operator/tuned-g95cv" Sep 10 13:17:22.280090 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:17:22.279848 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vcwwn\" (UniqueName: \"kubernetes.io/projected/bf875c9d-61a1-4b2f-8089-9b80ecb1c689-kube-api-access-vcwwn\") pod \"tuned-g95cv\" (UID: \"bf875c9d-61a1-4b2f-8089-9b80ecb1c689\") " pod="openshift-cluster-node-tuning-operator/tuned-g95cv" Sep 10 13:17:22.280090 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:17:22.279872 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/bf875c9d-61a1-4b2f-8089-9b80ecb1c689-run\") pod \"tuned-g95cv\" (UID: \"bf875c9d-61a1-4b2f-8089-9b80ecb1c689\") " pod="openshift-cluster-node-tuning-operator/tuned-g95cv" Sep 10 13:17:22.280090 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:17:22.279925 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/bf875c9d-61a1-4b2f-8089-9b80ecb1c689-etc-tuned\") pod \"tuned-g95cv\" (UID: \"bf875c9d-61a1-4b2f-8089-9b80ecb1c689\") " pod="openshift-cluster-node-tuning-operator/tuned-g95cv" Sep 10 13:17:22.280090 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:17:22.279955 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/eb907ce0-50ee-4062-9c00-5ede30885af1-device-dir\") pod \"aws-ebs-csi-driver-node-qnjfm\" (UID: \"eb907ce0-50ee-4062-9c00-5ede30885af1\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qnjfm" Sep 10 13:17:22.280090 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:17:22.279985 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7kdmf\" (UniqueName: \"kubernetes.io/projected/eb907ce0-50ee-4062-9c00-5ede30885af1-kube-api-access-7kdmf\") pod \"aws-ebs-csi-driver-node-qnjfm\" (UID: \"eb907ce0-50ee-4062-9c00-5ede30885af1\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qnjfm" Sep 10 13:17:22.280090 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:17:22.280015 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/2873d0b1-1e1d-4c45-a663-5b25c9308427-cnibin\") pod \"multus-additional-cni-plugins-sctq4\" (UID: \"2873d0b1-1e1d-4c45-a663-5b25c9308427\") " pod="openshift-multus/multus-additional-cni-plugins-sctq4" Sep 10 13:17:22.280090 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:17:22.280045 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/2873d0b1-1e1d-4c45-a663-5b25c9308427-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-sctq4\" (UID: \"2873d0b1-1e1d-4c45-a663-5b25c9308427\") " pod="openshift-multus/multus-additional-cni-plugins-sctq4" Sep 10 13:17:22.280090 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:17:22.280076 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/bf875c9d-61a1-4b2f-8089-9b80ecb1c689-etc-modprobe-d\") pod \"tuned-g95cv\" (UID: \"bf875c9d-61a1-4b2f-8089-9b80ecb1c689\") " pod="openshift-cluster-node-tuning-operator/tuned-g95cv" Sep 10 13:17:22.280532 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:17:22.280104 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/bf875c9d-61a1-4b2f-8089-9b80ecb1c689-etc-sysctl-conf\") pod \"tuned-g95cv\" (UID: \"bf875c9d-61a1-4b2f-8089-9b80ecb1c689\") " pod="openshift-cluster-node-tuning-operator/tuned-g95cv" Sep 10 13:17:22.280532 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:17:22.280131 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/bf875c9d-61a1-4b2f-8089-9b80ecb1c689-etc-systemd\") pod \"tuned-g95cv\" (UID: \"bf875c9d-61a1-4b2f-8089-9b80ecb1c689\") " pod="openshift-cluster-node-tuning-operator/tuned-g95cv" Sep 10 13:17:22.280532 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:17:22.280161 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/eb907ce0-50ee-4062-9c00-5ede30885af1-sys-fs\") pod \"aws-ebs-csi-driver-node-qnjfm\" (UID: \"eb907ce0-50ee-4062-9c00-5ede30885af1\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qnjfm" Sep 10 13:17:22.280532 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:17:22.280187 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/2873d0b1-1e1d-4c45-a663-5b25c9308427-system-cni-dir\") pod \"multus-additional-cni-plugins-sctq4\" (UID: \"2873d0b1-1e1d-4c45-a663-5b25c9308427\") " pod="openshift-multus/multus-additional-cni-plugins-sctq4" Sep 10 13:17:22.280532 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:17:22.280213 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/bf875c9d-61a1-4b2f-8089-9b80ecb1c689-etc-sysconfig\") pod \"tuned-g95cv\" (UID: \"bf875c9d-61a1-4b2f-8089-9b80ecb1c689\") " pod="openshift-cluster-node-tuning-operator/tuned-g95cv" Sep 10 13:17:22.280532 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:17:22.280235 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/bf875c9d-61a1-4b2f-8089-9b80ecb1c689-etc-sysctl-d\") pod \"tuned-g95cv\" (UID: \"bf875c9d-61a1-4b2f-8089-9b80ecb1c689\") " pod="openshift-cluster-node-tuning-operator/tuned-g95cv" Sep 10 13:17:22.280532 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:17:22.280257 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/bf875c9d-61a1-4b2f-8089-9b80ecb1c689-sys\") pod \"tuned-g95cv\" (UID: \"bf875c9d-61a1-4b2f-8089-9b80ecb1c689\") " pod="openshift-cluster-node-tuning-operator/tuned-g95cv" Sep 10 13:17:22.280532 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:17:22.280281 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/bf875c9d-61a1-4b2f-8089-9b80ecb1c689-lib-modules\") pod \"tuned-g95cv\" (UID: \"bf875c9d-61a1-4b2f-8089-9b80ecb1c689\") " pod="openshift-cluster-node-tuning-operator/tuned-g95cv" Sep 10 13:17:22.280532 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:17:22.280304 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/eb907ce0-50ee-4062-9c00-5ede30885af1-etc-selinux\") pod \"aws-ebs-csi-driver-node-qnjfm\" (UID: \"eb907ce0-50ee-4062-9c00-5ede30885af1\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qnjfm" Sep 10 13:17:22.280532 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:17:22.280341 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/2873d0b1-1e1d-4c45-a663-5b25c9308427-cni-binary-copy\") pod \"multus-additional-cni-plugins-sctq4\" (UID: \"2873d0b1-1e1d-4c45-a663-5b25c9308427\") " pod="openshift-multus/multus-additional-cni-plugins-sctq4" Sep 10 13:17:22.280532 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:17:22.280391 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/2873d0b1-1e1d-4c45-a663-5b25c9308427-tuning-conf-dir\") pod \"multus-additional-cni-plugins-sctq4\" (UID: \"2873d0b1-1e1d-4c45-a663-5b25c9308427\") " pod="openshift-multus/multus-additional-cni-plugins-sctq4" Sep 10 13:17:22.280532 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:17:22.280435 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/2873d0b1-1e1d-4c45-a663-5b25c9308427-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-sctq4\" (UID: \"2873d0b1-1e1d-4c45-a663-5b25c9308427\") " pod="openshift-multus/multus-additional-cni-plugins-sctq4" Sep 10 13:17:22.280532 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:17:22.280515 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/2873d0b1-1e1d-4c45-a663-5b25c9308427-cnibin\") pod \"multus-additional-cni-plugins-sctq4\" (UID: \"2873d0b1-1e1d-4c45-a663-5b25c9308427\") " pod="openshift-multus/multus-additional-cni-plugins-sctq4" Sep 10 13:17:22.281132 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:17:22.280547 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/bf875c9d-61a1-4b2f-8089-9b80ecb1c689-sys\") pod \"tuned-g95cv\" (UID: \"bf875c9d-61a1-4b2f-8089-9b80ecb1c689\") " pod="openshift-cluster-node-tuning-operator/tuned-g95cv" Sep 10 13:17:22.281132 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:17:22.280552 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/bf875c9d-61a1-4b2f-8089-9b80ecb1c689-etc-sysconfig\") pod \"tuned-g95cv\" (UID: \"bf875c9d-61a1-4b2f-8089-9b80ecb1c689\") " pod="openshift-cluster-node-tuning-operator/tuned-g95cv" Sep 10 13:17:22.281132 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:17:22.280669 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/bf875c9d-61a1-4b2f-8089-9b80ecb1c689-lib-modules\") pod \"tuned-g95cv\" (UID: \"bf875c9d-61a1-4b2f-8089-9b80ecb1c689\") " pod="openshift-cluster-node-tuning-operator/tuned-g95cv" Sep 10 13:17:22.281132 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:17:22.280748 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/eb907ce0-50ee-4062-9c00-5ede30885af1-etc-selinux\") pod \"aws-ebs-csi-driver-node-qnjfm\" (UID: \"eb907ce0-50ee-4062-9c00-5ede30885af1\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qnjfm" Sep 10 13:17:22.281132 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:17:22.280794 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/bf875c9d-61a1-4b2f-8089-9b80ecb1c689-etc-kubernetes\") pod \"tuned-g95cv\" (UID: \"bf875c9d-61a1-4b2f-8089-9b80ecb1c689\") " pod="openshift-cluster-node-tuning-operator/tuned-g95cv" Sep 10 13:17:22.281132 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:17:22.280851 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/bf875c9d-61a1-4b2f-8089-9b80ecb1c689-var-lib-kubelet\") pod \"tuned-g95cv\" (UID: \"bf875c9d-61a1-4b2f-8089-9b80ecb1c689\") " pod="openshift-cluster-node-tuning-operator/tuned-g95cv" Sep 10 13:17:22.281132 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:17:22.280879 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/bf875c9d-61a1-4b2f-8089-9b80ecb1c689-etc-modprobe-d\") pod \"tuned-g95cv\" (UID: \"bf875c9d-61a1-4b2f-8089-9b80ecb1c689\") " pod="openshift-cluster-node-tuning-operator/tuned-g95cv" Sep 10 13:17:22.281132 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:17:22.280970 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/bf875c9d-61a1-4b2f-8089-9b80ecb1c689-etc-sysctl-conf\") pod \"tuned-g95cv\" (UID: \"bf875c9d-61a1-4b2f-8089-9b80ecb1c689\") " pod="openshift-cluster-node-tuning-operator/tuned-g95cv" Sep 10 13:17:22.281132 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:17:22.281009 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/bf875c9d-61a1-4b2f-8089-9b80ecb1c689-etc-systemd\") pod \"tuned-g95cv\" (UID: \"bf875c9d-61a1-4b2f-8089-9b80ecb1c689\") " pod="openshift-cluster-node-tuning-operator/tuned-g95cv" Sep 10 13:17:22.281132 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:17:22.281042 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/eb907ce0-50ee-4062-9c00-5ede30885af1-sys-fs\") pod \"aws-ebs-csi-driver-node-qnjfm\" (UID: \"eb907ce0-50ee-4062-9c00-5ede30885af1\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qnjfm" Sep 10 13:17:22.281132 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:17:22.281066 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/2873d0b1-1e1d-4c45-a663-5b25c9308427-system-cni-dir\") pod \"multus-additional-cni-plugins-sctq4\" (UID: \"2873d0b1-1e1d-4c45-a663-5b25c9308427\") " pod="openshift-multus/multus-additional-cni-plugins-sctq4" Sep 10 13:17:22.281657 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:17:22.281186 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/2873d0b1-1e1d-4c45-a663-5b25c9308427-cni-binary-copy\") pod \"multus-additional-cni-plugins-sctq4\" (UID: \"2873d0b1-1e1d-4c45-a663-5b25c9308427\") " pod="openshift-multus/multus-additional-cni-plugins-sctq4" Sep 10 13:17:22.281657 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:17:22.281298 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/2873d0b1-1e1d-4c45-a663-5b25c9308427-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-sctq4\" (UID: \"2873d0b1-1e1d-4c45-a663-5b25c9308427\") " pod="openshift-multus/multus-additional-cni-plugins-sctq4" Sep 10 13:17:22.281657 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:17:22.281364 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/bf875c9d-61a1-4b2f-8089-9b80ecb1c689-host\") pod \"tuned-g95cv\" (UID: \"bf875c9d-61a1-4b2f-8089-9b80ecb1c689\") " pod="openshift-cluster-node-tuning-operator/tuned-g95cv" Sep 10 13:17:22.281657 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:17:22.281410 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/eb907ce0-50ee-4062-9c00-5ede30885af1-kubelet-dir\") pod \"aws-ebs-csi-driver-node-qnjfm\" (UID: \"eb907ce0-50ee-4062-9c00-5ede30885af1\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qnjfm" Sep 10 13:17:22.281657 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:17:22.281437 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/2873d0b1-1e1d-4c45-a663-5b25c9308427-tuning-conf-dir\") pod \"multus-additional-cni-plugins-sctq4\" (UID: \"2873d0b1-1e1d-4c45-a663-5b25c9308427\") " pod="openshift-multus/multus-additional-cni-plugins-sctq4" Sep 10 13:17:22.281657 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:17:22.281519 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/2873d0b1-1e1d-4c45-a663-5b25c9308427-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-sctq4\" (UID: \"2873d0b1-1e1d-4c45-a663-5b25c9308427\") " pod="openshift-multus/multus-additional-cni-plugins-sctq4" Sep 10 13:17:22.281657 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:17:22.281526 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/eb907ce0-50ee-4062-9c00-5ede30885af1-socket-dir\") pod \"aws-ebs-csi-driver-node-qnjfm\" (UID: \"eb907ce0-50ee-4062-9c00-5ede30885af1\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qnjfm" Sep 10 13:17:22.281657 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:17:22.281576 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/eb907ce0-50ee-4062-9c00-5ede30885af1-registration-dir\") pod \"aws-ebs-csi-driver-node-qnjfm\" (UID: \"eb907ce0-50ee-4062-9c00-5ede30885af1\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qnjfm" Sep 10 13:17:22.281657 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:17:22.281619 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/bf875c9d-61a1-4b2f-8089-9b80ecb1c689-etc-sysctl-d\") pod \"tuned-g95cv\" (UID: \"bf875c9d-61a1-4b2f-8089-9b80ecb1c689\") " pod="openshift-cluster-node-tuning-operator/tuned-g95cv" Sep 10 13:17:22.282282 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:17:22.282250 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/eb907ce0-50ee-4062-9c00-5ede30885af1-device-dir\") pod \"aws-ebs-csi-driver-node-qnjfm\" (UID: \"eb907ce0-50ee-4062-9c00-5ede30885af1\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qnjfm" Sep 10 13:17:22.282388 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:17:22.282358 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/bf875c9d-61a1-4b2f-8089-9b80ecb1c689-run\") pod \"tuned-g95cv\" (UID: \"bf875c9d-61a1-4b2f-8089-9b80ecb1c689\") " pod="openshift-cluster-node-tuning-operator/tuned-g95cv" Sep 10 13:17:22.284520 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:17:22.284496 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/bf875c9d-61a1-4b2f-8089-9b80ecb1c689-etc-tuned\") pod \"tuned-g95cv\" (UID: \"bf875c9d-61a1-4b2f-8089-9b80ecb1c689\") " pod="openshift-cluster-node-tuning-operator/tuned-g95cv" Sep 10 13:17:22.284632 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:17:22.284519 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/bf875c9d-61a1-4b2f-8089-9b80ecb1c689-tmp\") pod \"tuned-g95cv\" (UID: \"bf875c9d-61a1-4b2f-8089-9b80ecb1c689\") " pod="openshift-cluster-node-tuning-operator/tuned-g95cv" Sep 10 13:17:22.292460 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:17:22.292388 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7kdmf\" (UniqueName: \"kubernetes.io/projected/eb907ce0-50ee-4062-9c00-5ede30885af1-kube-api-access-7kdmf\") pod \"aws-ebs-csi-driver-node-qnjfm\" (UID: \"eb907ce0-50ee-4062-9c00-5ede30885af1\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qnjfm" Sep 10 13:17:22.293529 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:17:22.293507 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vcwwn\" (UniqueName: \"kubernetes.io/projected/bf875c9d-61a1-4b2f-8089-9b80ecb1c689-kube-api-access-vcwwn\") pod \"tuned-g95cv\" (UID: \"bf875c9d-61a1-4b2f-8089-9b80ecb1c689\") " pod="openshift-cluster-node-tuning-operator/tuned-g95cv" Sep 10 13:17:22.294243 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:17:22.294220 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tc5wf\" (UniqueName: \"kubernetes.io/projected/2873d0b1-1e1d-4c45-a663-5b25c9308427-kube-api-access-tc5wf\") pod \"multus-additional-cni-plugins-sctq4\" (UID: \"2873d0b1-1e1d-4c45-a663-5b25c9308427\") " pod="openshift-multus/multus-additional-cni-plugins-sctq4" Sep 10 13:17:22.372265 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:17:22.372234 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-kfhgq" Sep 10 13:17:22.383378 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:17:22.383340 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-6wf8v" Sep 10 13:17:22.392167 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:17:22.392148 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-zzkks" Sep 10 13:17:22.397942 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:17:22.397924 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-92n4q" Sep 10 13:17:22.405534 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:17:22.405516 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-65w2f" Sep 10 13:17:22.421107 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:17:22.421084 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-c22v2" Sep 10 13:17:22.430955 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:17:22.430937 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-sctq4" Sep 10 13:17:22.436609 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:17:22.436589 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qnjfm" Sep 10 13:17:22.443347 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:17:22.443330 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-g95cv" Sep 10 13:17:22.565711 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:17:22.565633 2557 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160" Sep 10 13:17:22.684774 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:17:22.684744 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4016c772-d4cc-4de5-9272-f33e9da5b2b8-metrics-certs\") pod \"network-metrics-daemon-7vvkj\" (UID: \"4016c772-d4cc-4de5-9272-f33e9da5b2b8\") " pod="openshift-multus/network-metrics-daemon-7vvkj" Sep 10 13:17:22.684928 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:17:22.684794 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/d873f3a0-ccac-4d59-a6f6-2c93cf5118de-original-pull-secret\") pod \"global-pull-secret-syncer-phs5r\" (UID: \"d873f3a0-ccac-4d59-a6f6-2c93cf5118de\") " pod="kube-system/global-pull-secret-syncer-phs5r" Sep 10 13:17:22.684928 ip-10-0-139-174 kubenswrapper[2557]: E0910 13:17:22.684912 2557 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Sep 10 13:17:22.685010 ip-10-0-139-174 kubenswrapper[2557]: E0910 13:17:22.684933 2557 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Sep 10 13:17:22.685010 ip-10-0-139-174 kubenswrapper[2557]: E0910 13:17:22.684973 2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d873f3a0-ccac-4d59-a6f6-2c93cf5118de-original-pull-secret podName:d873f3a0-ccac-4d59-a6f6-2c93cf5118de nodeName:}" failed. No retries permitted until 2026-09-10 13:17:23.68495013 +0000 UTC m=+3.991944522 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/d873f3a0-ccac-4d59-a6f6-2c93cf5118de-original-pull-secret") pod "global-pull-secret-syncer-phs5r" (UID: "d873f3a0-ccac-4d59-a6f6-2c93cf5118de") : object "kube-system"/"original-pull-secret" not registered Sep 10 13:17:22.685010 ip-10-0-139-174 kubenswrapper[2557]: E0910 13:17:22.685010 2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4016c772-d4cc-4de5-9272-f33e9da5b2b8-metrics-certs podName:4016c772-d4cc-4de5-9272-f33e9da5b2b8 nodeName:}" failed. No retries permitted until 2026-09-10 13:17:23.684992795 +0000 UTC m=+3.991987206 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/4016c772-d4cc-4de5-9272-f33e9da5b2b8-metrics-certs") pod "network-metrics-daemon-7vvkj" (UID: "4016c772-d4cc-4de5-9272-f33e9da5b2b8") : object "openshift-multus"/"metrics-daemon-secret" not registered Sep 10 13:17:22.785984 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:17:22.785952 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r6pw5\" (UniqueName: \"kubernetes.io/projected/accbb486-a5c9-469d-beca-837d2bbd4eab-kube-api-access-r6pw5\") pod \"network-check-target-6ggtx\" (UID: \"accbb486-a5c9-469d-beca-837d2bbd4eab\") " pod="openshift-network-diagnostics/network-check-target-6ggtx" Sep 10 13:17:22.786178 ip-10-0-139-174 kubenswrapper[2557]: E0910 13:17:22.786158 2557 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Sep 10 13:17:22.786229 ip-10-0-139-174 kubenswrapper[2557]: E0910 13:17:22.786185 2557 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Sep 10 13:17:22.786229 ip-10-0-139-174 kubenswrapper[2557]: E0910 13:17:22.786199 2557 projected.go:196] Error preparing data for projected volume kube-api-access-r6pw5 for pod openshift-network-diagnostics/network-check-target-6ggtx: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 10 13:17:22.786309 ip-10-0-139-174 kubenswrapper[2557]: E0910 13:17:22.786271 2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/accbb486-a5c9-469d-beca-837d2bbd4eab-kube-api-access-r6pw5 podName:accbb486-a5c9-469d-beca-837d2bbd4eab nodeName:}" failed. No retries permitted until 2026-09-10 13:17:23.786251824 +0000 UTC m=+4.093246231 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-r6pw5" (UniqueName: "kubernetes.io/projected/accbb486-a5c9-469d-beca-837d2bbd4eab-kube-api-access-r6pw5") pod "network-check-target-6ggtx" (UID: "accbb486-a5c9-469d-beca-837d2bbd4eab") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 10 13:17:23.114438 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:17:23.114359 2557 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-09-09 13:12:22 +0000 UTC" deadline="2028-04-03 16:11:03.054780889 +0000 UTC" Sep 10 13:17:23.114438 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:17:23.114413 2557 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13706h53m39.940370532s" Sep 10 13:17:23.214202 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:17:23.214178 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-7vvkj" Sep 10 13:17:23.214369 ip-10-0-139-174 kubenswrapper[2557]: E0910 13:17:23.214292 2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-7vvkj" podUID="4016c772-d4cc-4de5-9272-f33e9da5b2b8" Sep 10 13:17:23.214369 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:17:23.214359 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-phs5r" Sep 10 13:17:23.214492 ip-10-0-139-174 kubenswrapper[2557]: E0910 13:17:23.214472 2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-phs5r" podUID="d873f3a0-ccac-4d59-a6f6-2c93cf5118de" Sep 10 13:17:23.355204 ip-10-0-139-174 kubenswrapper[2557]: W0910 13:17:23.355177 2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod720acda6_0b15_49ac_ab4e_60de70c4324a.slice/crio-4b752d9f68927ce6d315bbf246100a8e7f11bb4e072cd6cd7961d33fdb329a01 WatchSource:0}: Error finding container 4b752d9f68927ce6d315bbf246100a8e7f11bb4e072cd6cd7961d33fdb329a01: Status 404 returned error can't find the container with id 4b752d9f68927ce6d315bbf246100a8e7f11bb4e072cd6cd7961d33fdb329a01 Sep 10 13:17:23.356951 ip-10-0-139-174 kubenswrapper[2557]: W0910 13:17:23.356914 2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9e7f9da8_1931_42da_b5c4_67542de73659.slice/crio-5921adb957dbc3724fe1cbbf004cef2aac0c985f9aa578507beb6fe4b6582aee WatchSource:0}: Error finding container 5921adb957dbc3724fe1cbbf004cef2aac0c985f9aa578507beb6fe4b6582aee: Status 404 returned error can't find the container with id 5921adb957dbc3724fe1cbbf004cef2aac0c985f9aa578507beb6fe4b6582aee Sep 10 13:17:23.359921 ip-10-0-139-174 kubenswrapper[2557]: W0910 13:17:23.359896 2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2873d0b1_1e1d_4c45_a663_5b25c9308427.slice/crio-ff587d323dde9658414954b036bbd82e170a4aaec42417c8628d0fa791a2fafe WatchSource:0}: Error finding container ff587d323dde9658414954b036bbd82e170a4aaec42417c8628d0fa791a2fafe: Status 404 returned error can't find the container with id ff587d323dde9658414954b036bbd82e170a4aaec42417c8628d0fa791a2fafe Sep 10 13:17:23.360533 ip-10-0-139-174 kubenswrapper[2557]: W0910 13:17:23.360495 2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8fe2ceee_5166_41b0_a39c_5732ea7e336b.slice/crio-be657c1de736b474b56ef5656a95a50feaf53b558c2a23edcc169364d7dafb8a WatchSource:0}: Error finding container be657c1de736b474b56ef5656a95a50feaf53b558c2a23edcc169364d7dafb8a: Status 404 returned error can't find the container with id be657c1de736b474b56ef5656a95a50feaf53b558c2a23edcc169364d7dafb8a Sep 10 13:17:23.381460 ip-10-0-139-174 kubenswrapper[2557]: W0910 13:17:23.381438 2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda13f63dc_c50e_4f46_84fe_d9718d8b2308.slice/crio-ef506a418deb53b3b29d9b3a119a0a4fa5c51cd00132c65397dd82f17a4aada2 WatchSource:0}: Error finding container ef506a418deb53b3b29d9b3a119a0a4fa5c51cd00132c65397dd82f17a4aada2: Status 404 returned error can't find the container with id ef506a418deb53b3b29d9b3a119a0a4fa5c51cd00132c65397dd82f17a4aada2 Sep 10 13:17:23.382204 ip-10-0-139-174 kubenswrapper[2557]: W0910 13:17:23.382106 2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod981d9727_9493_4d26_ac17_5fae22a92e0d.slice/crio-0d7d7751abc986f160cc1006dd2246159a804bcfd97eec4f1202c237da47dfd1 WatchSource:0}: Error finding container 0d7d7751abc986f160cc1006dd2246159a804bcfd97eec4f1202c237da47dfd1: Status 404 returned error can't find the container with id 0d7d7751abc986f160cc1006dd2246159a804bcfd97eec4f1202c237da47dfd1 Sep 10 13:17:23.383117 ip-10-0-139-174 kubenswrapper[2557]: W0910 13:17:23.382958 2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf2904741_875c_40e6_a97e_e56ae66d5077.slice/crio-3b51482b01212ed4e2c9dac5c0d705f0f9165d3f381388753ad900d1e6515312 WatchSource:0}: Error finding container 3b51482b01212ed4e2c9dac5c0d705f0f9165d3f381388753ad900d1e6515312: Status 404 returned error can't find the container with id 3b51482b01212ed4e2c9dac5c0d705f0f9165d3f381388753ad900d1e6515312 Sep 10 13:17:23.384013 ip-10-0-139-174 kubenswrapper[2557]: W0910 13:17:23.383987 2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbf875c9d_61a1_4b2f_8089_9b80ecb1c689.slice/crio-5263e1a432fe267bfb28f589b8f6f99bbb78a60ee0fc2ff82beaff896f3d9e8d WatchSource:0}: Error finding container 5263e1a432fe267bfb28f589b8f6f99bbb78a60ee0fc2ff82beaff896f3d9e8d: Status 404 returned error can't find the container with id 5263e1a432fe267bfb28f589b8f6f99bbb78a60ee0fc2ff82beaff896f3d9e8d Sep 10 13:17:23.386364 ip-10-0-139-174 kubenswrapper[2557]: W0910 13:17:23.386330 2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podeb907ce0_50ee_4062_9c00_5ede30885af1.slice/crio-29f0290390993324e9806cfd1b5cb7d98f930f3f6271bd453490ab0f51a7132c WatchSource:0}: Error finding container 29f0290390993324e9806cfd1b5cb7d98f930f3f6271bd453490ab0f51a7132c: Status 404 returned error can't find the container with id 29f0290390993324e9806cfd1b5cb7d98f930f3f6271bd453490ab0f51a7132c Sep 10 13:17:23.692076 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:17:23.692052 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4016c772-d4cc-4de5-9272-f33e9da5b2b8-metrics-certs\") pod \"network-metrics-daemon-7vvkj\" (UID: \"4016c772-d4cc-4de5-9272-f33e9da5b2b8\") " pod="openshift-multus/network-metrics-daemon-7vvkj" Sep 10 13:17:23.692187 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:17:23.692087 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/d873f3a0-ccac-4d59-a6f6-2c93cf5118de-original-pull-secret\") pod \"global-pull-secret-syncer-phs5r\" (UID: \"d873f3a0-ccac-4d59-a6f6-2c93cf5118de\") " pod="kube-system/global-pull-secret-syncer-phs5r" Sep 10 13:17:23.692250 ip-10-0-139-174 kubenswrapper[2557]: E0910 13:17:23.692185 2557 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Sep 10 13:17:23.692250 ip-10-0-139-174 kubenswrapper[2557]: E0910 13:17:23.692208 2557 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Sep 10 13:17:23.692250 ip-10-0-139-174 kubenswrapper[2557]: E0910 13:17:23.692237 2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d873f3a0-ccac-4d59-a6f6-2c93cf5118de-original-pull-secret podName:d873f3a0-ccac-4d59-a6f6-2c93cf5118de nodeName:}" failed. No retries permitted until 2026-09-10 13:17:25.69222487 +0000 UTC m=+5.999219262 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/d873f3a0-ccac-4d59-a6f6-2c93cf5118de-original-pull-secret") pod "global-pull-secret-syncer-phs5r" (UID: "d873f3a0-ccac-4d59-a6f6-2c93cf5118de") : object "kube-system"/"original-pull-secret" not registered Sep 10 13:17:23.692384 ip-10-0-139-174 kubenswrapper[2557]: E0910 13:17:23.692269 2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4016c772-d4cc-4de5-9272-f33e9da5b2b8-metrics-certs podName:4016c772-d4cc-4de5-9272-f33e9da5b2b8 nodeName:}" failed. No retries permitted until 2026-09-10 13:17:25.69225097 +0000 UTC m=+5.999245362 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/4016c772-d4cc-4de5-9272-f33e9da5b2b8-metrics-certs") pod "network-metrics-daemon-7vvkj" (UID: "4016c772-d4cc-4de5-9272-f33e9da5b2b8") : object "openshift-multus"/"metrics-daemon-secret" not registered Sep 10 13:17:23.762590 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:17:23.762564 2557 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160" Sep 10 13:17:23.792617 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:17:23.792588 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r6pw5\" (UniqueName: \"kubernetes.io/projected/accbb486-a5c9-469d-beca-837d2bbd4eab-kube-api-access-r6pw5\") pod \"network-check-target-6ggtx\" (UID: \"accbb486-a5c9-469d-beca-837d2bbd4eab\") " pod="openshift-network-diagnostics/network-check-target-6ggtx" Sep 10 13:17:23.792800 ip-10-0-139-174 kubenswrapper[2557]: E0910 13:17:23.792771 2557 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Sep 10 13:17:23.792800 ip-10-0-139-174 kubenswrapper[2557]: E0910 13:17:23.792804 2557 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Sep 10 13:17:23.792940 ip-10-0-139-174 kubenswrapper[2557]: E0910 13:17:23.792814 2557 projected.go:196] Error preparing data for projected volume kube-api-access-r6pw5 for pod openshift-network-diagnostics/network-check-target-6ggtx: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 10 13:17:23.792940 ip-10-0-139-174 kubenswrapper[2557]: E0910 13:17:23.792868 2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/accbb486-a5c9-469d-beca-837d2bbd4eab-kube-api-access-r6pw5 podName:accbb486-a5c9-469d-beca-837d2bbd4eab nodeName:}" failed. No retries permitted until 2026-09-10 13:17:25.792849091 +0000 UTC m=+6.099843487 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-r6pw5" (UniqueName: "kubernetes.io/projected/accbb486-a5c9-469d-beca-837d2bbd4eab-kube-api-access-r6pw5") pod "network-check-target-6ggtx" (UID: "accbb486-a5c9-469d-beca-837d2bbd4eab") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 10 13:17:24.115273 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:17:24.115202 2557 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-09-09 13:12:22 +0000 UTC" deadline="2028-04-12 19:25:33.003919997 +0000 UTC" Sep 10 13:17:24.115273 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:17:24.115241 2557 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13926h8m8.888684131s" Sep 10 13:17:24.217169 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:17:24.216443 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-6ggtx" Sep 10 13:17:24.217169 ip-10-0-139-174 kubenswrapper[2557]: E0910 13:17:24.216568 2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-6ggtx" podUID="accbb486-a5c9-469d-beca-837d2bbd4eab" Sep 10 13:17:24.243450 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:17:24.243103 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-139-174.ec2.internal" event={"ID":"b2d5d97b9a9b3ad7edee5cc4d2502cd5","Type":"ContainerStarted","Data":"84b0733e7d7e80e999a5404205202534ac7fedeac280d715877a8e15a0d490ce"} Sep 10 13:17:24.251027 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:17:24.250994 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qnjfm" event={"ID":"eb907ce0-50ee-4062-9c00-5ede30885af1","Type":"ContainerStarted","Data":"29f0290390993324e9806cfd1b5cb7d98f930f3f6271bd453490ab0f51a7132c"} Sep 10 13:17:24.255239 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:17:24.255209 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-zzkks" event={"ID":"9e7f9da8-1931-42da-b5c4-67542de73659","Type":"ContainerStarted","Data":"5921adb957dbc3724fe1cbbf004cef2aac0c985f9aa578507beb6fe4b6582aee"} Sep 10 13:17:24.260370 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:17:24.260316 2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-139-174.ec2.internal" podStartSLOduration=3.260299181 podStartE2EDuration="3.260299181s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-10 13:17:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 13:17:24.259377735 +0000 UTC m=+4.566372149" watchObservedRunningTime="2026-09-10 13:17:24.260299181 +0000 UTC m=+4.567293597" Sep 10 13:17:24.266368 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:17:24.266345 2557 generic.go:356] "Generic (PLEG): container finished" podID="329bcfe776c7597f50776992405be196" containerID="0e99faf0cc7c28f783787402dfa6f1aac15fa7b568d7c33ca5fff3c1b815583e" exitCode=0 Sep 10 13:17:24.266486 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:17:24.266445 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-174.ec2.internal" event={"ID":"329bcfe776c7597f50776992405be196","Type":"ContainerDied","Data":"0e99faf0cc7c28f783787402dfa6f1aac15fa7b568d7c33ca5fff3c1b815583e"} Sep 10 13:17:24.272345 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:17:24.272277 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-6wf8v" event={"ID":"f2904741-875c-40e6-a97e-e56ae66d5077","Type":"ContainerStarted","Data":"3b51482b01212ed4e2c9dac5c0d705f0f9165d3f381388753ad900d1e6515312"} Sep 10 13:17:24.274620 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:17:24.274561 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-g95cv" event={"ID":"bf875c9d-61a1-4b2f-8089-9b80ecb1c689","Type":"ContainerStarted","Data":"5263e1a432fe267bfb28f589b8f6f99bbb78a60ee0fc2ff82beaff896f3d9e8d"} Sep 10 13:17:24.300020 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:17:24.299809 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-c22v2" event={"ID":"981d9727-9493-4d26-ac17-5fae22a92e0d","Type":"ContainerStarted","Data":"0d7d7751abc986f160cc1006dd2246159a804bcfd97eec4f1202c237da47dfd1"} Sep 10 13:17:24.309749 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:17:24.309721 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-kfhgq" event={"ID":"a13f63dc-c50e-4f46-84fe-d9718d8b2308","Type":"ContainerStarted","Data":"ef506a418deb53b3b29d9b3a119a0a4fa5c51cd00132c65397dd82f17a4aada2"} Sep 10 13:17:24.313181 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:17:24.313157 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-65w2f" event={"ID":"8fe2ceee-5166-41b0-a39c-5732ea7e336b","Type":"ContainerStarted","Data":"be657c1de736b474b56ef5656a95a50feaf53b558c2a23edcc169364d7dafb8a"} Sep 10 13:17:24.317024 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:17:24.316998 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-sctq4" event={"ID":"2873d0b1-1e1d-4c45-a663-5b25c9308427","Type":"ContainerStarted","Data":"ff587d323dde9658414954b036bbd82e170a4aaec42417c8628d0fa791a2fafe"} Sep 10 13:17:24.319175 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:17:24.319147 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-92n4q" event={"ID":"720acda6-0b15-49ac-ab4e-60de70c4324a","Type":"ContainerStarted","Data":"4b752d9f68927ce6d315bbf246100a8e7f11bb4e072cd6cd7961d33fdb329a01"} Sep 10 13:17:25.214030 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:17:25.213993 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-7vvkj" Sep 10 13:17:25.214658 ip-10-0-139-174 kubenswrapper[2557]: E0910 13:17:25.214134 2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-7vvkj" podUID="4016c772-d4cc-4de5-9272-f33e9da5b2b8" Sep 10 13:17:25.214658 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:17:25.214530 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-phs5r" Sep 10 13:17:25.214658 ip-10-0-139-174 kubenswrapper[2557]: E0910 13:17:25.214621 2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-phs5r" podUID="d873f3a0-ccac-4d59-a6f6-2c93cf5118de" Sep 10 13:17:25.335037 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:17:25.334961 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-174.ec2.internal" event={"ID":"329bcfe776c7597f50776992405be196","Type":"ContainerStarted","Data":"6a30fd96b697973c9d7777327c3dfc6182bbb1ead2d801221822e9b99d27c515"} Sep 10 13:17:25.708996 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:17:25.708912 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/d873f3a0-ccac-4d59-a6f6-2c93cf5118de-original-pull-secret\") pod \"global-pull-secret-syncer-phs5r\" (UID: \"d873f3a0-ccac-4d59-a6f6-2c93cf5118de\") " pod="kube-system/global-pull-secret-syncer-phs5r" Sep 10 13:17:25.709163 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:17:25.709014 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4016c772-d4cc-4de5-9272-f33e9da5b2b8-metrics-certs\") pod \"network-metrics-daemon-7vvkj\" (UID: \"4016c772-d4cc-4de5-9272-f33e9da5b2b8\") " pod="openshift-multus/network-metrics-daemon-7vvkj" Sep 10 13:17:25.709163 ip-10-0-139-174 kubenswrapper[2557]: E0910 13:17:25.709145 2557 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Sep 10 13:17:25.709275 ip-10-0-139-174 kubenswrapper[2557]: E0910 13:17:25.709208 2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4016c772-d4cc-4de5-9272-f33e9da5b2b8-metrics-certs podName:4016c772-d4cc-4de5-9272-f33e9da5b2b8 nodeName:}" failed. No retries permitted until 2026-09-10 13:17:29.709188708 +0000 UTC m=+10.016183107 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/4016c772-d4cc-4de5-9272-f33e9da5b2b8-metrics-certs") pod "network-metrics-daemon-7vvkj" (UID: "4016c772-d4cc-4de5-9272-f33e9da5b2b8") : object "openshift-multus"/"metrics-daemon-secret" not registered Sep 10 13:17:25.709597 ip-10-0-139-174 kubenswrapper[2557]: E0910 13:17:25.709572 2557 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Sep 10 13:17:25.709687 ip-10-0-139-174 kubenswrapper[2557]: E0910 13:17:25.709640 2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d873f3a0-ccac-4d59-a6f6-2c93cf5118de-original-pull-secret podName:d873f3a0-ccac-4d59-a6f6-2c93cf5118de nodeName:}" failed. No retries permitted until 2026-09-10 13:17:29.709622976 +0000 UTC m=+10.016617369 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/d873f3a0-ccac-4d59-a6f6-2c93cf5118de-original-pull-secret") pod "global-pull-secret-syncer-phs5r" (UID: "d873f3a0-ccac-4d59-a6f6-2c93cf5118de") : object "kube-system"/"original-pull-secret" not registered Sep 10 13:17:25.809555 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:17:25.809518 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r6pw5\" (UniqueName: \"kubernetes.io/projected/accbb486-a5c9-469d-beca-837d2bbd4eab-kube-api-access-r6pw5\") pod \"network-check-target-6ggtx\" (UID: \"accbb486-a5c9-469d-beca-837d2bbd4eab\") " pod="openshift-network-diagnostics/network-check-target-6ggtx" Sep 10 13:17:25.809729 ip-10-0-139-174 kubenswrapper[2557]: E0910 13:17:25.809708 2557 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Sep 10 13:17:25.809800 ip-10-0-139-174 kubenswrapper[2557]: E0910 13:17:25.809732 2557 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Sep 10 13:17:25.809800 ip-10-0-139-174 kubenswrapper[2557]: E0910 13:17:25.809744 2557 projected.go:196] Error preparing data for projected volume kube-api-access-r6pw5 for pod openshift-network-diagnostics/network-check-target-6ggtx: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 10 13:17:25.809904 ip-10-0-139-174 kubenswrapper[2557]: E0910 13:17:25.809801 2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/accbb486-a5c9-469d-beca-837d2bbd4eab-kube-api-access-r6pw5 podName:accbb486-a5c9-469d-beca-837d2bbd4eab nodeName:}" failed. No retries permitted until 2026-09-10 13:17:29.809782545 +0000 UTC m=+10.116776952 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-r6pw5" (UniqueName: "kubernetes.io/projected/accbb486-a5c9-469d-beca-837d2bbd4eab-kube-api-access-r6pw5") pod "network-check-target-6ggtx" (UID: "accbb486-a5c9-469d-beca-837d2bbd4eab") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 10 13:17:26.216984 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:17:26.216951 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-6ggtx" Sep 10 13:17:26.217441 ip-10-0-139-174 kubenswrapper[2557]: E0910 13:17:26.217087 2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-6ggtx" podUID="accbb486-a5c9-469d-beca-837d2bbd4eab" Sep 10 13:17:27.214186 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:17:27.214126 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-phs5r" Sep 10 13:17:27.214376 ip-10-0-139-174 kubenswrapper[2557]: E0910 13:17:27.214245 2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-phs5r" podUID="d873f3a0-ccac-4d59-a6f6-2c93cf5118de" Sep 10 13:17:27.214616 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:17:27.214121 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-7vvkj" Sep 10 13:17:27.214720 ip-10-0-139-174 kubenswrapper[2557]: E0910 13:17:27.214704 2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-7vvkj" podUID="4016c772-d4cc-4de5-9272-f33e9da5b2b8" Sep 10 13:17:28.217026 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:17:28.216449 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-6ggtx" Sep 10 13:17:28.217026 ip-10-0-139-174 kubenswrapper[2557]: E0910 13:17:28.216604 2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-6ggtx" podUID="accbb486-a5c9-469d-beca-837d2bbd4eab" Sep 10 13:17:29.214002 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:17:29.213971 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-7vvkj" Sep 10 13:17:29.214002 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:17:29.213996 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-phs5r" Sep 10 13:17:29.214242 ip-10-0-139-174 kubenswrapper[2557]: E0910 13:17:29.214089 2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-phs5r" podUID="d873f3a0-ccac-4d59-a6f6-2c93cf5118de" Sep 10 13:17:29.214242 ip-10-0-139-174 kubenswrapper[2557]: E0910 13:17:29.214202 2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-7vvkj" podUID="4016c772-d4cc-4de5-9272-f33e9da5b2b8" Sep 10 13:17:29.749859 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:17:29.749820 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/d873f3a0-ccac-4d59-a6f6-2c93cf5118de-original-pull-secret\") pod \"global-pull-secret-syncer-phs5r\" (UID: \"d873f3a0-ccac-4d59-a6f6-2c93cf5118de\") " pod="kube-system/global-pull-secret-syncer-phs5r" Sep 10 13:17:29.750245 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:17:29.749905 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4016c772-d4cc-4de5-9272-f33e9da5b2b8-metrics-certs\") pod \"network-metrics-daemon-7vvkj\" (UID: \"4016c772-d4cc-4de5-9272-f33e9da5b2b8\") " pod="openshift-multus/network-metrics-daemon-7vvkj" Sep 10 13:17:29.750245 ip-10-0-139-174 kubenswrapper[2557]: E0910 13:17:29.750005 2557 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Sep 10 13:17:29.750245 ip-10-0-139-174 kubenswrapper[2557]: E0910 13:17:29.750081 2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4016c772-d4cc-4de5-9272-f33e9da5b2b8-metrics-certs podName:4016c772-d4cc-4de5-9272-f33e9da5b2b8 nodeName:}" failed. No retries permitted until 2026-09-10 13:17:37.750061582 +0000 UTC m=+18.057055978 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/4016c772-d4cc-4de5-9272-f33e9da5b2b8-metrics-certs") pod "network-metrics-daemon-7vvkj" (UID: "4016c772-d4cc-4de5-9272-f33e9da5b2b8") : object "openshift-multus"/"metrics-daemon-secret" not registered Sep 10 13:17:29.750507 ip-10-0-139-174 kubenswrapper[2557]: E0910 13:17:29.750486 2557 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Sep 10 13:17:29.750597 ip-10-0-139-174 kubenswrapper[2557]: E0910 13:17:29.750542 2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d873f3a0-ccac-4d59-a6f6-2c93cf5118de-original-pull-secret podName:d873f3a0-ccac-4d59-a6f6-2c93cf5118de nodeName:}" failed. No retries permitted until 2026-09-10 13:17:37.750525386 +0000 UTC m=+18.057519809 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/d873f3a0-ccac-4d59-a6f6-2c93cf5118de-original-pull-secret") pod "global-pull-secret-syncer-phs5r" (UID: "d873f3a0-ccac-4d59-a6f6-2c93cf5118de") : object "kube-system"/"original-pull-secret" not registered Sep 10 13:17:29.851470 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:17:29.850778 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r6pw5\" (UniqueName: \"kubernetes.io/projected/accbb486-a5c9-469d-beca-837d2bbd4eab-kube-api-access-r6pw5\") pod \"network-check-target-6ggtx\" (UID: \"accbb486-a5c9-469d-beca-837d2bbd4eab\") " pod="openshift-network-diagnostics/network-check-target-6ggtx" Sep 10 13:17:29.851470 ip-10-0-139-174 kubenswrapper[2557]: E0910 13:17:29.850981 2557 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Sep 10 13:17:29.851470 ip-10-0-139-174 kubenswrapper[2557]: E0910 13:17:29.851002 2557 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Sep 10 13:17:29.851470 ip-10-0-139-174 kubenswrapper[2557]: E0910 13:17:29.851014 2557 projected.go:196] Error preparing data for projected volume kube-api-access-r6pw5 for pod openshift-network-diagnostics/network-check-target-6ggtx: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 10 13:17:29.851470 ip-10-0-139-174 kubenswrapper[2557]: E0910 13:17:29.851075 2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/accbb486-a5c9-469d-beca-837d2bbd4eab-kube-api-access-r6pw5 podName:accbb486-a5c9-469d-beca-837d2bbd4eab nodeName:}" failed. No retries permitted until 2026-09-10 13:17:37.851057034 +0000 UTC m=+18.158051438 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-r6pw5" (UniqueName: "kubernetes.io/projected/accbb486-a5c9-469d-beca-837d2bbd4eab-kube-api-access-r6pw5") pod "network-check-target-6ggtx" (UID: "accbb486-a5c9-469d-beca-837d2bbd4eab") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 10 13:17:30.214982 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:17:30.214948 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-6ggtx" Sep 10 13:17:30.215136 ip-10-0-139-174 kubenswrapper[2557]: E0910 13:17:30.215030 2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-6ggtx" podUID="accbb486-a5c9-469d-beca-837d2bbd4eab" Sep 10 13:17:31.214283 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:17:31.214241 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-7vvkj" Sep 10 13:17:31.214779 ip-10-0-139-174 kubenswrapper[2557]: E0910 13:17:31.214373 2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-7vvkj" podUID="4016c772-d4cc-4de5-9272-f33e9da5b2b8" Sep 10 13:17:31.214779 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:17:31.214691 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-phs5r" Sep 10 13:17:31.214779 ip-10-0-139-174 kubenswrapper[2557]: E0910 13:17:31.214750 2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-phs5r" podUID="d873f3a0-ccac-4d59-a6f6-2c93cf5118de" Sep 10 13:17:32.216436 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:17:32.216380 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-6ggtx" Sep 10 13:17:32.216878 ip-10-0-139-174 kubenswrapper[2557]: E0910 13:17:32.216524 2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-6ggtx" podUID="accbb486-a5c9-469d-beca-837d2bbd4eab" Sep 10 13:17:33.214145 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:17:33.213655 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-7vvkj" Sep 10 13:17:33.214145 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:17:33.213703 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-phs5r" Sep 10 13:17:33.214145 ip-10-0-139-174 kubenswrapper[2557]: E0910 13:17:33.213805 2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-7vvkj" podUID="4016c772-d4cc-4de5-9272-f33e9da5b2b8" Sep 10 13:17:33.214145 ip-10-0-139-174 kubenswrapper[2557]: E0910 13:17:33.213947 2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-phs5r" podUID="d873f3a0-ccac-4d59-a6f6-2c93cf5118de" Sep 10 13:17:34.214050 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:17:34.214016 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-6ggtx" Sep 10 13:17:34.214481 ip-10-0-139-174 kubenswrapper[2557]: E0910 13:17:34.214141 2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-6ggtx" podUID="accbb486-a5c9-469d-beca-837d2bbd4eab" Sep 10 13:17:35.213789 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:17:35.213748 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-7vvkj" Sep 10 13:17:35.213943 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:17:35.213795 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-phs5r" Sep 10 13:17:35.213943 ip-10-0-139-174 kubenswrapper[2557]: E0910 13:17:35.213877 2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-7vvkj" podUID="4016c772-d4cc-4de5-9272-f33e9da5b2b8" Sep 10 13:17:35.214031 ip-10-0-139-174 kubenswrapper[2557]: E0910 13:17:35.213992 2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-phs5r" podUID="d873f3a0-ccac-4d59-a6f6-2c93cf5118de" Sep 10 13:17:36.213984 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:17:36.213953 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-6ggtx" Sep 10 13:17:36.214436 ip-10-0-139-174 kubenswrapper[2557]: E0910 13:17:36.214082 2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-6ggtx" podUID="accbb486-a5c9-469d-beca-837d2bbd4eab" Sep 10 13:17:37.213357 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:17:37.213322 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-phs5r" Sep 10 13:17:37.213524 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:17:37.213364 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-7vvkj" Sep 10 13:17:37.213524 ip-10-0-139-174 kubenswrapper[2557]: E0910 13:17:37.213461 2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-phs5r" podUID="d873f3a0-ccac-4d59-a6f6-2c93cf5118de" Sep 10 13:17:37.213633 ip-10-0-139-174 kubenswrapper[2557]: E0910 13:17:37.213594 2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-7vvkj" podUID="4016c772-d4cc-4de5-9272-f33e9da5b2b8" Sep 10 13:17:37.809472 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:17:37.809436 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4016c772-d4cc-4de5-9272-f33e9da5b2b8-metrics-certs\") pod \"network-metrics-daemon-7vvkj\" (UID: \"4016c772-d4cc-4de5-9272-f33e9da5b2b8\") " pod="openshift-multus/network-metrics-daemon-7vvkj" Sep 10 13:17:37.809902 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:17:37.809483 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/d873f3a0-ccac-4d59-a6f6-2c93cf5118de-original-pull-secret\") pod \"global-pull-secret-syncer-phs5r\" (UID: \"d873f3a0-ccac-4d59-a6f6-2c93cf5118de\") " pod="kube-system/global-pull-secret-syncer-phs5r" Sep 10 13:17:37.809902 ip-10-0-139-174 kubenswrapper[2557]: E0910 13:17:37.809610 2557 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Sep 10 13:17:37.809902 ip-10-0-139-174 kubenswrapper[2557]: E0910 13:17:37.809611 2557 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Sep 10 13:17:37.809902 ip-10-0-139-174 kubenswrapper[2557]: E0910 13:17:37.809683 2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d873f3a0-ccac-4d59-a6f6-2c93cf5118de-original-pull-secret podName:d873f3a0-ccac-4d59-a6f6-2c93cf5118de nodeName:}" failed. No retries permitted until 2026-09-10 13:17:53.809663632 +0000 UTC m=+34.116658038 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/d873f3a0-ccac-4d59-a6f6-2c93cf5118de-original-pull-secret") pod "global-pull-secret-syncer-phs5r" (UID: "d873f3a0-ccac-4d59-a6f6-2c93cf5118de") : object "kube-system"/"original-pull-secret" not registered Sep 10 13:17:37.809902 ip-10-0-139-174 kubenswrapper[2557]: E0910 13:17:37.809703 2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4016c772-d4cc-4de5-9272-f33e9da5b2b8-metrics-certs podName:4016c772-d4cc-4de5-9272-f33e9da5b2b8 nodeName:}" failed. No retries permitted until 2026-09-10 13:17:53.809692837 +0000 UTC m=+34.116687235 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/4016c772-d4cc-4de5-9272-f33e9da5b2b8-metrics-certs") pod "network-metrics-daemon-7vvkj" (UID: "4016c772-d4cc-4de5-9272-f33e9da5b2b8") : object "openshift-multus"/"metrics-daemon-secret" not registered Sep 10 13:17:37.910326 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:17:37.910298 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r6pw5\" (UniqueName: \"kubernetes.io/projected/accbb486-a5c9-469d-beca-837d2bbd4eab-kube-api-access-r6pw5\") pod \"network-check-target-6ggtx\" (UID: \"accbb486-a5c9-469d-beca-837d2bbd4eab\") " pod="openshift-network-diagnostics/network-check-target-6ggtx" Sep 10 13:17:37.910495 ip-10-0-139-174 kubenswrapper[2557]: E0910 13:17:37.910443 2557 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Sep 10 13:17:37.910495 ip-10-0-139-174 kubenswrapper[2557]: E0910 13:17:37.910460 2557 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Sep 10 13:17:37.910495 ip-10-0-139-174 kubenswrapper[2557]: E0910 13:17:37.910469 2557 projected.go:196] Error preparing data for projected volume kube-api-access-r6pw5 for pod openshift-network-diagnostics/network-check-target-6ggtx: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 10 13:17:37.910639 ip-10-0-139-174 kubenswrapper[2557]: E0910 13:17:37.910541 2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/accbb486-a5c9-469d-beca-837d2bbd4eab-kube-api-access-r6pw5 podName:accbb486-a5c9-469d-beca-837d2bbd4eab nodeName:}" failed. No retries permitted until 2026-09-10 13:17:53.91052335 +0000 UTC m=+34.217517754 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-r6pw5" (UniqueName: "kubernetes.io/projected/accbb486-a5c9-469d-beca-837d2bbd4eab-kube-api-access-r6pw5") pod "network-check-target-6ggtx" (UID: "accbb486-a5c9-469d-beca-837d2bbd4eab") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 10 13:17:38.213464 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:17:38.213433 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-6ggtx" Sep 10 13:17:38.213635 ip-10-0-139-174 kubenswrapper[2557]: E0910 13:17:38.213555 2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-6ggtx" podUID="accbb486-a5c9-469d-beca-837d2bbd4eab" Sep 10 13:17:39.213659 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:17:39.213627 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-7vvkj" Sep 10 13:17:39.213659 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:17:39.213661 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-phs5r" Sep 10 13:17:39.214137 ip-10-0-139-174 kubenswrapper[2557]: E0910 13:17:39.213744 2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-7vvkj" podUID="4016c772-d4cc-4de5-9272-f33e9da5b2b8" Sep 10 13:17:39.214137 ip-10-0-139-174 kubenswrapper[2557]: E0910 13:17:39.213826 2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-phs5r" podUID="d873f3a0-ccac-4d59-a6f6-2c93cf5118de" Sep 10 13:17:40.214760 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:17:40.214721 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-6ggtx" Sep 10 13:17:40.215191 ip-10-0-139-174 kubenswrapper[2557]: E0910 13:17:40.214839 2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-6ggtx" podUID="accbb486-a5c9-469d-beca-837d2bbd4eab" Sep 10 13:17:41.214148 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:17:41.214121 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-phs5r" Sep 10 13:17:41.214148 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:17:41.214150 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-7vvkj" Sep 10 13:17:41.214319 ip-10-0-139-174 kubenswrapper[2557]: E0910 13:17:41.214213 2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-phs5r" podUID="d873f3a0-ccac-4d59-a6f6-2c93cf5118de" Sep 10 13:17:41.214363 ip-10-0-139-174 kubenswrapper[2557]: E0910 13:17:41.214339 2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-7vvkj" podUID="4016c772-d4cc-4de5-9272-f33e9da5b2b8" Sep 10 13:17:41.373192 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:17:41.373164 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-kfhgq" event={"ID":"a13f63dc-c50e-4f46-84fe-d9718d8b2308","Type":"ContainerStarted","Data":"7f0b82d8766518ab63244b00a780973bc9f020aabddd7e00c9f3a97c94e1abfa"} Sep 10 13:17:41.409398 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:17:41.409347 2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-174.ec2.internal" podStartSLOduration=20.409326325 podStartE2EDuration="20.409326325s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-10 13:17:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 13:17:25.354798686 +0000 UTC m=+5.661793102" watchObservedRunningTime="2026-09-10 13:17:41.409326325 +0000 UTC m=+21.716320742" Sep 10 13:17:42.214380 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:17:42.214208 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-6ggtx" Sep 10 13:17:42.214586 ip-10-0-139-174 kubenswrapper[2557]: E0910 13:17:42.214503 2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-6ggtx" podUID="accbb486-a5c9-469d-beca-837d2bbd4eab" Sep 10 13:17:42.377234 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:17:42.376999 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qnjfm" event={"ID":"eb907ce0-50ee-4062-9c00-5ede30885af1","Type":"ContainerStarted","Data":"783a6ab941655a08a41fb534cc95195ff30421ebcc00a180f648411e12e97f1e"} Sep 10 13:17:42.378926 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:17:42.378893 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-zzkks" event={"ID":"9e7f9da8-1931-42da-b5c4-67542de73659","Type":"ContainerStarted","Data":"28c7f5dbeb6a270ace0b2d91c6e51a1d5bc2c4a026146cfb6b5f970cfb6931c0"} Sep 10 13:17:42.380747 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:17:42.380719 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-6wf8v" event={"ID":"f2904741-875c-40e6-a97e-e56ae66d5077","Type":"ContainerStarted","Data":"193faa2273dbdfd1c5a9aa946e059736889a87f897aeb56c4bedc3a35cf5cdab"} Sep 10 13:17:42.382569 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:17:42.382537 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-g95cv" event={"ID":"bf875c9d-61a1-4b2f-8089-9b80ecb1c689","Type":"ContainerStarted","Data":"0e466ef1ef1c90949834e2dbbb18f2d0529002cc6e6b4977dfdb51a89850ec32"} Sep 10 13:17:42.384234 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:17:42.384206 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-c22v2" event={"ID":"981d9727-9493-4d26-ac17-5fae22a92e0d","Type":"ContainerStarted","Data":"0b0734b1dd2995fbfb42266b74ab48a8744a82c96b94d768745dce4f8be3bc91"} Sep 10 13:17:42.386057 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:17:42.386035 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-65w2f" event={"ID":"8fe2ceee-5166-41b0-a39c-5732ea7e336b","Type":"ContainerStarted","Data":"abf36724e2e6f3bb2ec8aaeb00610eec33c648d8f081c9d54e3bebf9f2a67563"} Sep 10 13:17:42.388087 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:17:42.388064 2557 generic.go:356] "Generic (PLEG): container finished" podID="2873d0b1-1e1d-4c45-a663-5b25c9308427" containerID="9128af9adefa4f5d4e97de09f3bf1209c4af9d13487a06cd0caf07d4b0ffbccc" exitCode=0 Sep 10 13:17:42.388171 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:17:42.388128 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-sctq4" event={"ID":"2873d0b1-1e1d-4c45-a663-5b25c9308427","Type":"ContainerDied","Data":"9128af9adefa4f5d4e97de09f3bf1209c4af9d13487a06cd0caf07d4b0ffbccc"} Sep 10 13:17:42.392580 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:17:42.392560 2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-92n4q_720acda6-0b15-49ac-ab4e-60de70c4324a/ovn-acl-logging/0.log" Sep 10 13:17:42.392957 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:17:42.392938 2557 generic.go:356] "Generic (PLEG): container finished" podID="720acda6-0b15-49ac-ab4e-60de70c4324a" containerID="68efaf8607b2588ce9d50abf062142bea2e428b32ad0bd5ae9e58f7b58bc8457" exitCode=1 Sep 10 13:17:42.393026 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:17:42.393013 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-92n4q" event={"ID":"720acda6-0b15-49ac-ab4e-60de70c4324a","Type":"ContainerStarted","Data":"0766d7b08cd7296f9af7220f9bc9a65d64b531e58b3958370e92ce2cc4c44e68"} Sep 10 13:17:42.393115 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:17:42.393080 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-92n4q" event={"ID":"720acda6-0b15-49ac-ab4e-60de70c4324a","Type":"ContainerStarted","Data":"13456d844fcab5a491376dead7683a4120de67948f23a24b560078983b44610b"} Sep 10 13:17:42.393160 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:17:42.393132 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-92n4q" event={"ID":"720acda6-0b15-49ac-ab4e-60de70c4324a","Type":"ContainerStarted","Data":"ef2db854681b786308d05cdffc66d28728f6d22d25823069d6a97a6bf548ed3c"} Sep 10 13:17:42.393223 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:17:42.393194 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-92n4q" event={"ID":"720acda6-0b15-49ac-ab4e-60de70c4324a","Type":"ContainerStarted","Data":"89d10680fc5720381636088dbfe30061b1f895c39449c412e99b22abbb113801"} Sep 10 13:17:42.393281 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:17:42.393244 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-92n4q" event={"ID":"720acda6-0b15-49ac-ab4e-60de70c4324a","Type":"ContainerDied","Data":"68efaf8607b2588ce9d50abf062142bea2e428b32ad0bd5ae9e58f7b58bc8457"} Sep 10 13:17:42.393330 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:17:42.393296 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-92n4q" event={"ID":"720acda6-0b15-49ac-ab4e-60de70c4324a","Type":"ContainerStarted","Data":"c457f8a84a672858366f8742bfd6c4272d036f1d61e9f86e7eb2ec896c47f85d"} Sep 10 13:17:42.400781 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:17:42.400745 2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-kfhgq" podStartSLOduration=11.929448286 podStartE2EDuration="22.400733362s" totalImagesPullingTime="10.471285076s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 13:17:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 13:17:41.409224105 +0000 UTC m=+21.716218521" watchObservedRunningTime="2026-09-10 13:17:42.400733362 +0000 UTC m=+22.707727766" Sep 10 13:17:42.400968 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:17:42.400950 2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-zzkks" podStartSLOduration=4.529090243 podStartE2EDuration="22.400943667s" totalImagesPullingTime="17.871853424s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-10 13:17:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 13:17:42.400704355 +0000 UTC m=+22.707698770" watchObservedRunningTime="2026-09-10 13:17:42.400943667 +0000 UTC m=+22.707938080" Sep 10 13:17:42.416259 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:17:42.416223 2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-c22v2" podStartSLOduration=4.568585857 podStartE2EDuration="22.41621125s" totalImagesPullingTime="17.847625393s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-10 13:17:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 13:17:42.415889901 +0000 UTC m=+22.722884326" watchObservedRunningTime="2026-09-10 13:17:42.41621125 +0000 UTC m=+22.723205664" Sep 10 13:17:42.454598 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:17:42.454558 2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-6wf8v" podStartSLOduration=4.607611699 podStartE2EDuration="22.454546884s" totalImagesPullingTime="17.846935185s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-10 13:17:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 13:17:42.454216116 +0000 UTC m=+22.761210529" watchObservedRunningTime="2026-09-10 13:17:42.454546884 +0000 UTC m=+22.761541298" Sep 10 13:17:42.475875 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:17:42.475756 2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-65w2f" podStartSLOduration=4.580545419 podStartE2EDuration="22.475735775s" totalImagesPullingTime="17.895190356s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-10 13:17:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 13:17:42.475585582 +0000 UTC m=+22.782579995" watchObservedRunningTime="2026-09-10 13:17:42.475735775 +0000 UTC m=+22.782730190" Sep 10 13:17:42.477925 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:17:42.477907 2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-zzkks_9e7f9da8-1931-42da-b5c4-67542de73659/node-ca/0.log" Sep 10 13:17:42.942274 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:17:42.942245 2557 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock" Sep 10 13:17:43.134358 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:17:43.134197 2557 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-09-10T13:17:42.942269855Z","UUID":"fa57c8e3-cd04-4f06-84fb-522416f91191","Handler":null,"Name":"","Endpoint":""} Sep 10 13:17:43.135469 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:17:43.135453 2557 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0 Sep 10 13:17:43.135527 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:17:43.135480 2557 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock Sep 10 13:17:43.214303 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:17:43.214284 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-7vvkj" Sep 10 13:17:43.214400 ip-10-0-139-174 kubenswrapper[2557]: E0910 13:17:43.214372 2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-7vvkj" podUID="4016c772-d4cc-4de5-9272-f33e9da5b2b8" Sep 10 13:17:43.214463 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:17:43.214443 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-phs5r" Sep 10 13:17:43.214535 ip-10-0-139-174 kubenswrapper[2557]: E0910 13:17:43.214520 2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-phs5r" podUID="d873f3a0-ccac-4d59-a6f6-2c93cf5118de" Sep 10 13:17:43.397874 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:17:43.397752 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qnjfm" event={"ID":"eb907ce0-50ee-4062-9c00-5ede30885af1","Type":"ContainerStarted","Data":"c65ea7151f831a4226be31cb9cd7990f81759efe79259d29720ac92c1633b874"} Sep 10 13:17:43.738900 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:17:43.738872 2557 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-kfhgq" Sep 10 13:17:43.739577 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:17:43.739559 2557 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-kfhgq" Sep 10 13:17:43.763521 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:17:43.763478 2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-g95cv" podStartSLOduration=5.740689474 podStartE2EDuration="23.763459917s" totalImagesPullingTime="18.022770443s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-10 13:17:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 13:17:42.493297357 +0000 UTC m=+22.800291771" watchObservedRunningTime="2026-09-10 13:17:43.763459917 +0000 UTC m=+24.070454330" Sep 10 13:17:44.161098 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:17:44.161069 2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-kfhgq" Sep 10 13:17:44.161525 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:17:44.161508 2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-kfhgq" Sep 10 13:17:44.214227 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:17:44.214205 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-6ggtx" Sep 10 13:17:44.214344 ip-10-0-139-174 kubenswrapper[2557]: E0910 13:17:44.214322 2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-6ggtx" podUID="accbb486-a5c9-469d-beca-837d2bbd4eab" Sep 10 13:17:44.404732 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:17:44.404708 2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-92n4q_720acda6-0b15-49ac-ab4e-60de70c4324a/ovn-acl-logging/0.log" Sep 10 13:17:44.405779 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:17:44.405561 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-92n4q" event={"ID":"720acda6-0b15-49ac-ab4e-60de70c4324a","Type":"ContainerStarted","Data":"193daaef81962b5b6856aa41b17af2bd820ad6e0ad493f7eb0da00eab1dc242e"} Sep 10 13:17:45.214141 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:17:45.214106 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-7vvkj" Sep 10 13:17:45.214141 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:17:45.214127 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-phs5r" Sep 10 13:17:45.214342 ip-10-0-139-174 kubenswrapper[2557]: E0910 13:17:45.214224 2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-phs5r" podUID="d873f3a0-ccac-4d59-a6f6-2c93cf5118de" Sep 10 13:17:45.214380 ip-10-0-139-174 kubenswrapper[2557]: E0910 13:17:45.214343 2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-7vvkj" podUID="4016c772-d4cc-4de5-9272-f33e9da5b2b8" Sep 10 13:17:45.410454 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:17:45.410399 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qnjfm" event={"ID":"eb907ce0-50ee-4062-9c00-5ede30885af1","Type":"ContainerStarted","Data":"af813ddd9d096e4cd4f1ba6db10ec1f1049aef1c96133891b55702e9cd539b7f"} Sep 10 13:17:45.456087 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:17:45.456042 2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qnjfm" podStartSLOduration=4.921743252 podStartE2EDuration="25.456028344s" totalImagesPullingTime="20.534285092s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-10 13:17:20 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 13:17:45.455609937 +0000 UTC m=+25.762604351" watchObservedRunningTime="2026-09-10 13:17:45.456028344 +0000 UTC m=+25.763022757" Sep 10 13:17:46.214066 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:17:46.214030 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-6ggtx" Sep 10 13:17:46.214223 ip-10-0-139-174 kubenswrapper[2557]: E0910 13:17:46.214154 2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-6ggtx" podUID="accbb486-a5c9-469d-beca-837d2bbd4eab" Sep 10 13:17:47.213376 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:17:47.213344 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-7vvkj" Sep 10 13:17:47.213815 ip-10-0-139-174 kubenswrapper[2557]: E0910 13:17:47.213467 2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-7vvkj" podUID="4016c772-d4cc-4de5-9272-f33e9da5b2b8" Sep 10 13:17:47.213815 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:17:47.213531 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-phs5r" Sep 10 13:17:47.214174 ip-10-0-139-174 kubenswrapper[2557]: E0910 13:17:47.214145 2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-phs5r" podUID="d873f3a0-ccac-4d59-a6f6-2c93cf5118de" Sep 10 13:17:48.213628 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:17:48.213457 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-6ggtx" Sep 10 13:17:48.214281 ip-10-0-139-174 kubenswrapper[2557]: E0910 13:17:48.213728 2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-6ggtx" podUID="accbb486-a5c9-469d-beca-837d2bbd4eab" Sep 10 13:17:48.419581 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:17:48.419552 2557 generic.go:356] "Generic (PLEG): container finished" podID="2873d0b1-1e1d-4c45-a663-5b25c9308427" containerID="1980863fb5f9bf00581beaa2d3b597aca74e0820c998f559b8a6f551356a5917" exitCode=0 Sep 10 13:17:48.419791 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:17:48.419723 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-sctq4" event={"ID":"2873d0b1-1e1d-4c45-a663-5b25c9308427","Type":"ContainerDied","Data":"1980863fb5f9bf00581beaa2d3b597aca74e0820c998f559b8a6f551356a5917"} Sep 10 13:17:48.426020 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:17:48.426003 2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-92n4q_720acda6-0b15-49ac-ab4e-60de70c4324a/ovn-acl-logging/0.log" Sep 10 13:17:48.426364 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:17:48.426341 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-92n4q" event={"ID":"720acda6-0b15-49ac-ab4e-60de70c4324a","Type":"ContainerStarted","Data":"9776073bc3e4b0eb6a0d5355be135b4ce438122fc52562d11a8868c8c97d7357"} Sep 10 13:17:48.426619 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:17:48.426599 2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-92n4q" Sep 10 13:17:48.426797 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:17:48.426782 2557 scope.go:117] "RemoveContainer" containerID="68efaf8607b2588ce9d50abf062142bea2e428b32ad0bd5ae9e58f7b58bc8457" Sep 10 13:17:48.443104 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:17:48.443083 2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-92n4q" Sep 10 13:17:49.213543 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:17:49.213518 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-phs5r" Sep 10 13:17:49.213640 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:17:49.213527 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-7vvkj" Sep 10 13:17:49.213928 ip-10-0-139-174 kubenswrapper[2557]: E0910 13:17:49.213600 2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-phs5r" podUID="d873f3a0-ccac-4d59-a6f6-2c93cf5118de" Sep 10 13:17:49.213928 ip-10-0-139-174 kubenswrapper[2557]: E0910 13:17:49.213725 2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-7vvkj" podUID="4016c772-d4cc-4de5-9272-f33e9da5b2b8" Sep 10 13:17:49.225999 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:17:49.225977 2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-92n4q" Sep 10 13:17:49.430113 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:17:49.430047 2557 generic.go:356] "Generic (PLEG): container finished" podID="2873d0b1-1e1d-4c45-a663-5b25c9308427" containerID="45bc5491d94f466f096f127bbbc38c17f3b1b5ef9040ba7d59140e6691e5cd81" exitCode=0 Sep 10 13:17:49.430225 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:17:49.430120 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-sctq4" event={"ID":"2873d0b1-1e1d-4c45-a663-5b25c9308427","Type":"ContainerDied","Data":"45bc5491d94f466f096f127bbbc38c17f3b1b5ef9040ba7d59140e6691e5cd81"} Sep 10 13:17:49.434099 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:17:49.434080 2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-92n4q_720acda6-0b15-49ac-ab4e-60de70c4324a/ovn-acl-logging/0.log" Sep 10 13:17:49.434509 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:17:49.434488 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-92n4q" event={"ID":"720acda6-0b15-49ac-ab4e-60de70c4324a","Type":"ContainerStarted","Data":"578ee49b721b739af13392c3df69a27836cdc1b7b3a1ba5e35fe3cb23ad8c218"} Sep 10 13:17:49.434696 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:17:49.434678 2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-92n4q" Sep 10 13:17:49.450035 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:17:49.450018 2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-92n4q" Sep 10 13:17:50.166953 ip-10-0-139-174 kubenswrapper[2557]: E0910 13:17:50.166788 2557 cadvisor_stats_provider.go:567] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2873d0b1_1e1d_4c45_a663_5b25c9308427.slice/crio-conmon-5bdf145963e65b803b9e2fe6195ed2b24c3606c1ced503e35c461236f4f59c45.scope\": RecentStats: unable to find data in memory cache]" Sep 10 13:17:50.214874 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:17:50.214853 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-6ggtx" Sep 10 13:17:50.215148 ip-10-0-139-174 kubenswrapper[2557]: E0910 13:17:50.214926 2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-6ggtx" podUID="accbb486-a5c9-469d-beca-837d2bbd4eab" Sep 10 13:17:50.438931 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:17:50.438849 2557 generic.go:356] "Generic (PLEG): container finished" podID="2873d0b1-1e1d-4c45-a663-5b25c9308427" containerID="5bdf145963e65b803b9e2fe6195ed2b24c3606c1ced503e35c461236f4f59c45" exitCode=0 Sep 10 13:17:50.439031 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:17:50.438922 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-sctq4" event={"ID":"2873d0b1-1e1d-4c45-a663-5b25c9308427","Type":"ContainerDied","Data":"5bdf145963e65b803b9e2fe6195ed2b24c3606c1ced503e35c461236f4f59c45"} Sep 10 13:17:50.459982 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:17:50.459947 2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-92n4q" podStartSLOduration=12.56749093 podStartE2EDuration="30.459934254s" totalImagesPullingTime="17.892443324s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-10 13:17:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 13:17:49.481978501 +0000 UTC m=+29.788972916" watchObservedRunningTime="2026-09-10 13:17:50.459934254 +0000 UTC m=+30.766928667" Sep 10 13:17:51.214156 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:17:51.214121 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-7vvkj" Sep 10 13:17:51.214291 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:17:51.214169 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-phs5r" Sep 10 13:17:51.214291 ip-10-0-139-174 kubenswrapper[2557]: E0910 13:17:51.214238 2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-7vvkj" podUID="4016c772-d4cc-4de5-9272-f33e9da5b2b8" Sep 10 13:17:51.214388 ip-10-0-139-174 kubenswrapper[2557]: E0910 13:17:51.214337 2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-phs5r" podUID="d873f3a0-ccac-4d59-a6f6-2c93cf5118de" Sep 10 13:17:52.213945 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:17:52.213913 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-6ggtx" Sep 10 13:17:52.214276 ip-10-0-139-174 kubenswrapper[2557]: E0910 13:17:52.214020 2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-6ggtx" podUID="accbb486-a5c9-469d-beca-837d2bbd4eab" Sep 10 13:17:53.214143 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:17:53.214101 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-7vvkj" Sep 10 13:17:53.214143 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:17:53.214136 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-phs5r" Sep 10 13:17:53.214699 ip-10-0-139-174 kubenswrapper[2557]: E0910 13:17:53.214226 2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-7vvkj" podUID="4016c772-d4cc-4de5-9272-f33e9da5b2b8" Sep 10 13:17:53.214699 ip-10-0-139-174 kubenswrapper[2557]: E0910 13:17:53.214380 2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-phs5r" podUID="d873f3a0-ccac-4d59-a6f6-2c93cf5118de" Sep 10 13:17:53.824002 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:17:53.823968 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4016c772-d4cc-4de5-9272-f33e9da5b2b8-metrics-certs\") pod \"network-metrics-daemon-7vvkj\" (UID: \"4016c772-d4cc-4de5-9272-f33e9da5b2b8\") " pod="openshift-multus/network-metrics-daemon-7vvkj" Sep 10 13:17:53.824208 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:17:53.824019 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/d873f3a0-ccac-4d59-a6f6-2c93cf5118de-original-pull-secret\") pod \"global-pull-secret-syncer-phs5r\" (UID: \"d873f3a0-ccac-4d59-a6f6-2c93cf5118de\") " pod="kube-system/global-pull-secret-syncer-phs5r" Sep 10 13:17:53.824208 ip-10-0-139-174 kubenswrapper[2557]: E0910 13:17:53.824134 2557 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Sep 10 13:17:53.824208 ip-10-0-139-174 kubenswrapper[2557]: E0910 13:17:53.824150 2557 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Sep 10 13:17:53.824208 ip-10-0-139-174 kubenswrapper[2557]: E0910 13:17:53.824209 2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d873f3a0-ccac-4d59-a6f6-2c93cf5118de-original-pull-secret podName:d873f3a0-ccac-4d59-a6f6-2c93cf5118de nodeName:}" failed. No retries permitted until 2026-09-10 13:18:25.824190743 +0000 UTC m=+66.131185134 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/d873f3a0-ccac-4d59-a6f6-2c93cf5118de-original-pull-secret") pod "global-pull-secret-syncer-phs5r" (UID: "d873f3a0-ccac-4d59-a6f6-2c93cf5118de") : object "kube-system"/"original-pull-secret" not registered Sep 10 13:17:53.824438 ip-10-0-139-174 kubenswrapper[2557]: E0910 13:17:53.824229 2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4016c772-d4cc-4de5-9272-f33e9da5b2b8-metrics-certs podName:4016c772-d4cc-4de5-9272-f33e9da5b2b8 nodeName:}" failed. No retries permitted until 2026-09-10 13:18:25.82421807 +0000 UTC m=+66.131212465 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/4016c772-d4cc-4de5-9272-f33e9da5b2b8-metrics-certs") pod "network-metrics-daemon-7vvkj" (UID: "4016c772-d4cc-4de5-9272-f33e9da5b2b8") : object "openshift-multus"/"metrics-daemon-secret" not registered Sep 10 13:17:53.924890 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:17:53.924863 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r6pw5\" (UniqueName: \"kubernetes.io/projected/accbb486-a5c9-469d-beca-837d2bbd4eab-kube-api-access-r6pw5\") pod \"network-check-target-6ggtx\" (UID: \"accbb486-a5c9-469d-beca-837d2bbd4eab\") " pod="openshift-network-diagnostics/network-check-target-6ggtx" Sep 10 13:17:53.925059 ip-10-0-139-174 kubenswrapper[2557]: E0910 13:17:53.925043 2557 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Sep 10 13:17:53.925123 ip-10-0-139-174 kubenswrapper[2557]: E0910 13:17:53.925063 2557 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Sep 10 13:17:53.925123 ip-10-0-139-174 kubenswrapper[2557]: E0910 13:17:53.925073 2557 projected.go:196] Error preparing data for projected volume kube-api-access-r6pw5 for pod openshift-network-diagnostics/network-check-target-6ggtx: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 10 13:17:53.925123 ip-10-0-139-174 kubenswrapper[2557]: E0910 13:17:53.925121 2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/accbb486-a5c9-469d-beca-837d2bbd4eab-kube-api-access-r6pw5 podName:accbb486-a5c9-469d-beca-837d2bbd4eab nodeName:}" failed. No retries permitted until 2026-09-10 13:18:25.925106302 +0000 UTC m=+66.232100711 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-r6pw5" (UniqueName: "kubernetes.io/projected/accbb486-a5c9-469d-beca-837d2bbd4eab-kube-api-access-r6pw5") pod "network-check-target-6ggtx" (UID: "accbb486-a5c9-469d-beca-837d2bbd4eab") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 10 13:17:54.213466 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:17:54.213432 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-6ggtx" Sep 10 13:17:54.213642 ip-10-0-139-174 kubenswrapper[2557]: E0910 13:17:54.213560 2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-6ggtx" podUID="accbb486-a5c9-469d-beca-837d2bbd4eab" Sep 10 13:17:55.213551 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:17:55.213526 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-7vvkj" Sep 10 13:17:55.213925 ip-10-0-139-174 kubenswrapper[2557]: E0910 13:17:55.213669 2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-7vvkj" podUID="4016c772-d4cc-4de5-9272-f33e9da5b2b8" Sep 10 13:17:55.213925 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:17:55.213696 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-phs5r" Sep 10 13:17:55.213925 ip-10-0-139-174 kubenswrapper[2557]: E0910 13:17:55.213817 2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-phs5r" podUID="d873f3a0-ccac-4d59-a6f6-2c93cf5118de" Sep 10 13:17:55.453348 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:17:55.453317 2557 generic.go:356] "Generic (PLEG): container finished" podID="2873d0b1-1e1d-4c45-a663-5b25c9308427" containerID="0ff54208a820fb123c7b81eb5f91781d0e97ab24a642d974f611c09d90866094" exitCode=0 Sep 10 13:17:55.453505 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:17:55.453375 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-sctq4" event={"ID":"2873d0b1-1e1d-4c45-a663-5b25c9308427","Type":"ContainerDied","Data":"0ff54208a820fb123c7b81eb5f91781d0e97ab24a642d974f611c09d90866094"} Sep 10 13:17:56.213674 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:17:56.213642 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-6ggtx" Sep 10 13:17:56.214015 ip-10-0-139-174 kubenswrapper[2557]: E0910 13:17:56.213763 2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-6ggtx" podUID="accbb486-a5c9-469d-beca-837d2bbd4eab" Sep 10 13:17:56.458721 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:17:56.458691 2557 generic.go:356] "Generic (PLEG): container finished" podID="2873d0b1-1e1d-4c45-a663-5b25c9308427" containerID="112564c2a47941483037517c80830f9b93a47ddd95d68be3cc68cbfd398af834" exitCode=0 Sep 10 13:17:56.458892 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:17:56.458760 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-sctq4" event={"ID":"2873d0b1-1e1d-4c45-a663-5b25c9308427","Type":"ContainerDied","Data":"112564c2a47941483037517c80830f9b93a47ddd95d68be3cc68cbfd398af834"} Sep 10 13:17:57.213702 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:17:57.213670 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-phs5r" Sep 10 13:17:57.214080 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:17:57.213670 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-7vvkj" Sep 10 13:17:57.214080 ip-10-0-139-174 kubenswrapper[2557]: E0910 13:17:57.213774 2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-phs5r" podUID="d873f3a0-ccac-4d59-a6f6-2c93cf5118de" Sep 10 13:17:57.214080 ip-10-0-139-174 kubenswrapper[2557]: E0910 13:17:57.213847 2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-7vvkj" podUID="4016c772-d4cc-4de5-9272-f33e9da5b2b8" Sep 10 13:17:57.463936 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:17:57.463866 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-sctq4" event={"ID":"2873d0b1-1e1d-4c45-a663-5b25c9308427","Type":"ContainerStarted","Data":"fd054c316b1da60a7b0670c39db3173711ffd1f9e8b9a7bde2d34cbae6254b49"} Sep 10 13:17:57.487920 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:17:57.487874 2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-sctq4" podStartSLOduration=8.384463337 podStartE2EDuration="37.487859594s" totalImagesPullingTime="29.103396257s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-10 13:17:20 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 13:17:57.48761559 +0000 UTC m=+37.794610004" watchObservedRunningTime="2026-09-10 13:17:57.487859594 +0000 UTC m=+37.794854008" Sep 10 13:17:58.214254 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:17:58.214227 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-6ggtx" Sep 10 13:17:58.214643 ip-10-0-139-174 kubenswrapper[2557]: E0910 13:17:58.214345 2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-6ggtx" podUID="accbb486-a5c9-469d-beca-837d2bbd4eab" Sep 10 13:17:59.213641 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:17:59.213608 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-7vvkj" Sep 10 13:17:59.213641 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:17:59.213638 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-phs5r" Sep 10 13:17:59.213830 ip-10-0-139-174 kubenswrapper[2557]: E0910 13:17:59.213733 2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-7vvkj" podUID="4016c772-d4cc-4de5-9272-f33e9da5b2b8" Sep 10 13:17:59.213871 ip-10-0-139-174 kubenswrapper[2557]: E0910 13:17:59.213841 2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-phs5r" podUID="d873f3a0-ccac-4d59-a6f6-2c93cf5118de" Sep 10 13:18:00.214356 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:18:00.214328 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-6ggtx" Sep 10 13:18:00.214698 ip-10-0-139-174 kubenswrapper[2557]: E0910 13:18:00.214405 2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-6ggtx" podUID="accbb486-a5c9-469d-beca-837d2bbd4eab" Sep 10 13:18:01.214268 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:18:01.214235 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-7vvkj" Sep 10 13:18:01.214455 ip-10-0-139-174 kubenswrapper[2557]: E0910 13:18:01.214328 2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-7vvkj" podUID="4016c772-d4cc-4de5-9272-f33e9da5b2b8" Sep 10 13:18:01.214455 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:18:01.214392 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-phs5r" Sep 10 13:18:01.214812 ip-10-0-139-174 kubenswrapper[2557]: E0910 13:18:01.214510 2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-phs5r" podUID="d873f3a0-ccac-4d59-a6f6-2c93cf5118de" Sep 10 13:18:02.213393 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:18:02.213365 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-6ggtx" Sep 10 13:18:02.213576 ip-10-0-139-174 kubenswrapper[2557]: E0910 13:18:02.213498 2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-6ggtx" podUID="accbb486-a5c9-469d-beca-837d2bbd4eab" Sep 10 13:18:03.213721 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:18:03.213690 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-7vvkj" Sep 10 13:18:03.214138 ip-10-0-139-174 kubenswrapper[2557]: E0910 13:18:03.213785 2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-7vvkj" podUID="4016c772-d4cc-4de5-9272-f33e9da5b2b8" Sep 10 13:18:03.214138 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:18:03.213857 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-phs5r" Sep 10 13:18:03.214138 ip-10-0-139-174 kubenswrapper[2557]: E0910 13:18:03.213964 2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-phs5r" podUID="d873f3a0-ccac-4d59-a6f6-2c93cf5118de" Sep 10 13:18:04.213792 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:18:04.213761 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-6ggtx" Sep 10 13:18:04.214201 ip-10-0-139-174 kubenswrapper[2557]: E0910 13:18:04.213872 2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-6ggtx" podUID="accbb486-a5c9-469d-beca-837d2bbd4eab" Sep 10 13:18:05.214095 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:18:05.214064 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-7vvkj" Sep 10 13:18:05.214556 ip-10-0-139-174 kubenswrapper[2557]: E0910 13:18:05.214188 2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-7vvkj" podUID="4016c772-d4cc-4de5-9272-f33e9da5b2b8" Sep 10 13:18:05.214556 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:18:05.214233 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-phs5r" Sep 10 13:18:05.214556 ip-10-0-139-174 kubenswrapper[2557]: E0910 13:18:05.214307 2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-phs5r" podUID="d873f3a0-ccac-4d59-a6f6-2c93cf5118de" Sep 10 13:18:06.213663 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:18:06.213634 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-6ggtx" Sep 10 13:18:06.213818 ip-10-0-139-174 kubenswrapper[2557]: E0910 13:18:06.213731 2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-6ggtx" podUID="accbb486-a5c9-469d-beca-837d2bbd4eab" Sep 10 13:18:07.213459 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:18:07.213413 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-7vvkj" Sep 10 13:18:07.213881 ip-10-0-139-174 kubenswrapper[2557]: E0910 13:18:07.213558 2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-7vvkj" podUID="4016c772-d4cc-4de5-9272-f33e9da5b2b8" Sep 10 13:18:07.213881 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:18:07.213608 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-phs5r" Sep 10 13:18:07.213881 ip-10-0-139-174 kubenswrapper[2557]: E0910 13:18:07.213684 2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-phs5r" podUID="d873f3a0-ccac-4d59-a6f6-2c93cf5118de" Sep 10 13:18:08.214031 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:18:08.213997 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-6ggtx" Sep 10 13:18:08.214490 ip-10-0-139-174 kubenswrapper[2557]: E0910 13:18:08.214090 2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-6ggtx" podUID="accbb486-a5c9-469d-beca-837d2bbd4eab" Sep 10 13:18:09.213890 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:18:09.213857 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-7vvkj" Sep 10 13:18:09.214127 ip-10-0-139-174 kubenswrapper[2557]: E0910 13:18:09.213959 2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-7vvkj" podUID="4016c772-d4cc-4de5-9272-f33e9da5b2b8" Sep 10 13:18:09.214127 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:18:09.214016 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-phs5r" Sep 10 13:18:09.214127 ip-10-0-139-174 kubenswrapper[2557]: E0910 13:18:09.214098 2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-phs5r" podUID="d873f3a0-ccac-4d59-a6f6-2c93cf5118de" Sep 10 13:18:10.214456 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:18:10.214406 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-6ggtx" Sep 10 13:18:10.214852 ip-10-0-139-174 kubenswrapper[2557]: E0910 13:18:10.214519 2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-6ggtx" podUID="accbb486-a5c9-469d-beca-837d2bbd4eab" Sep 10 13:18:11.213501 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:18:11.213464 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-phs5r" Sep 10 13:18:11.213501 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:18:11.213484 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-7vvkj" Sep 10 13:18:11.213722 ip-10-0-139-174 kubenswrapper[2557]: E0910 13:18:11.213562 2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-phs5r" podUID="d873f3a0-ccac-4d59-a6f6-2c93cf5118de" Sep 10 13:18:11.213722 ip-10-0-139-174 kubenswrapper[2557]: E0910 13:18:11.213700 2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-7vvkj" podUID="4016c772-d4cc-4de5-9272-f33e9da5b2b8" Sep 10 13:18:12.214089 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:18:12.214051 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-6ggtx" Sep 10 13:18:12.214539 ip-10-0-139-174 kubenswrapper[2557]: E0910 13:18:12.214151 2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-6ggtx" podUID="accbb486-a5c9-469d-beca-837d2bbd4eab" Sep 10 13:18:12.663398 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:18:12.663158 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-phs5r"] Sep 10 13:18:12.663592 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:18:12.663501 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-phs5r" Sep 10 13:18:12.663650 ip-10-0-139-174 kubenswrapper[2557]: E0910 13:18:12.663615 2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-phs5r" podUID="d873f3a0-ccac-4d59-a6f6-2c93cf5118de" Sep 10 13:18:12.664256 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:18:12.664206 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-6ggtx"] Sep 10 13:18:12.664434 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:18:12.664292 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-6ggtx" Sep 10 13:18:12.664434 ip-10-0-139-174 kubenswrapper[2557]: E0910 13:18:12.664372 2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-6ggtx" podUID="accbb486-a5c9-469d-beca-837d2bbd4eab" Sep 10 13:18:12.665037 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:18:12.665010 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-7vvkj"] Sep 10 13:18:12.665270 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:18:12.665242 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-7vvkj" Sep 10 13:18:12.665377 ip-10-0-139-174 kubenswrapper[2557]: E0910 13:18:12.665355 2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-7vvkj" podUID="4016c772-d4cc-4de5-9272-f33e9da5b2b8" Sep 10 13:18:14.214101 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:18:14.214063 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-6ggtx" Sep 10 13:18:14.214101 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:18:14.214089 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-phs5r" Sep 10 13:18:14.214580 ip-10-0-139-174 kubenswrapper[2557]: E0910 13:18:14.214176 2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-6ggtx" podUID="accbb486-a5c9-469d-beca-837d2bbd4eab" Sep 10 13:18:14.214580 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:18:14.214215 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-7vvkj" Sep 10 13:18:14.214580 ip-10-0-139-174 kubenswrapper[2557]: E0910 13:18:14.214344 2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-7vvkj" podUID="4016c772-d4cc-4de5-9272-f33e9da5b2b8" Sep 10 13:18:14.214580 ip-10-0-139-174 kubenswrapper[2557]: E0910 13:18:14.214468 2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-phs5r" podUID="d873f3a0-ccac-4d59-a6f6-2c93cf5118de" Sep 10 13:18:16.216559 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:18:16.216531 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-6ggtx" Sep 10 13:18:16.216993 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:18:16.216531 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-7vvkj" Sep 10 13:18:16.216993 ip-10-0-139-174 kubenswrapper[2557]: E0910 13:18:16.216623 2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-6ggtx" podUID="accbb486-a5c9-469d-beca-837d2bbd4eab" Sep 10 13:18:16.216993 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:18:16.216531 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-phs5r" Sep 10 13:18:16.216993 ip-10-0-139-174 kubenswrapper[2557]: E0910 13:18:16.216709 2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-7vvkj" podUID="4016c772-d4cc-4de5-9272-f33e9da5b2b8" Sep 10 13:18:16.216993 ip-10-0-139-174 kubenswrapper[2557]: E0910 13:18:16.216765 2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-phs5r" podUID="d873f3a0-ccac-4d59-a6f6-2c93cf5118de" Sep 10 13:18:16.456372 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:18:16.456347 2557 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-174.ec2.internal" event="NodeReady" Sep 10 13:18:16.456532 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:18:16.456453 2557 kubelet_node_status.go:489] "Fast updating node status as it just became ready" Sep 10 13:18:16.505248 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:18:16.505222 2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-9x5xs"] Sep 10 13:18:16.507385 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:18:16.507369 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-9x5xs" Sep 10 13:18:16.508255 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:18:16.508234 2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-m7dz6"] Sep 10 13:18:16.509944 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:18:16.509899 2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\"" Sep 10 13:18:16.509944 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:18:16.509923 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-fkjjg\"" Sep 10 13:18:16.510110 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:18:16.509961 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\"" Sep 10 13:18:16.510110 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:18:16.510066 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-m7dz6" Sep 10 13:18:16.511938 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:18:16.511919 2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\"" Sep 10 13:18:16.512027 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:18:16.511928 2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\"" Sep 10 13:18:16.512550 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:18:16.512538 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-74kfd\"" Sep 10 13:18:16.513375 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:18:16.513350 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\"" Sep 10 13:18:16.522845 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:18:16.522814 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-m7dz6"] Sep 10 13:18:16.523398 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:18:16.523380 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-9x5xs"] Sep 10 13:18:16.529922 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:18:16.529904 2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-7k285"] Sep 10 13:18:16.531951 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:18:16.531934 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-7k285" Sep 10 13:18:16.533893 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:18:16.533871 2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\"" Sep 10 13:18:16.533893 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:18:16.533889 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\"" Sep 10 13:18:16.534068 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:18:16.534013 2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\"" Sep 10 13:18:16.534202 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:18:16.534179 2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\"" Sep 10 13:18:16.534304 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:18:16.534258 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-rwkt7\"" Sep 10 13:18:16.544812 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:18:16.544792 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-7k285"] Sep 10 13:18:16.595628 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:18:16.595598 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/a5ede13b-c926-4b47-822c-384dc218e8f5-config-volume\") pod \"dns-default-9x5xs\" (UID: \"a5ede13b-c926-4b47-822c-384dc218e8f5\") " pod="openshift-dns/dns-default-9x5xs" Sep 10 13:18:16.595771 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:18:16.595630 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zs6lv\" (UniqueName: \"kubernetes.io/projected/a5ede13b-c926-4b47-822c-384dc218e8f5-kube-api-access-zs6lv\") pod \"dns-default-9x5xs\" (UID: \"a5ede13b-c926-4b47-822c-384dc218e8f5\") " pod="openshift-dns/dns-default-9x5xs" Sep 10 13:18:16.595771 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:18:16.595660 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/67184dde-9271-4b22-adb6-9a94adaf6c57-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-7k285\" (UID: \"67184dde-9271-4b22-adb6-9a94adaf6c57\") " pod="openshift-insights/insights-runtime-extractor-7k285" Sep 10 13:18:16.595771 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:18:16.595691 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/a5ede13b-c926-4b47-822c-384dc218e8f5-metrics-tls\") pod \"dns-default-9x5xs\" (UID: \"a5ede13b-c926-4b47-822c-384dc218e8f5\") " pod="openshift-dns/dns-default-9x5xs" Sep 10 13:18:16.595771 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:18:16.595720 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/67184dde-9271-4b22-adb6-9a94adaf6c57-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-7k285\" (UID: \"67184dde-9271-4b22-adb6-9a94adaf6c57\") " pod="openshift-insights/insights-runtime-extractor-7k285" Sep 10 13:18:16.595771 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:18:16.595749 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/67184dde-9271-4b22-adb6-9a94adaf6c57-crio-socket\") pod \"insights-runtime-extractor-7k285\" (UID: \"67184dde-9271-4b22-adb6-9a94adaf6c57\") " pod="openshift-insights/insights-runtime-extractor-7k285" Sep 10 13:18:16.595941 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:18:16.595775 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5dfgh\" (UniqueName: \"kubernetes.io/projected/67184dde-9271-4b22-adb6-9a94adaf6c57-kube-api-access-5dfgh\") pod \"insights-runtime-extractor-7k285\" (UID: \"67184dde-9271-4b22-adb6-9a94adaf6c57\") " pod="openshift-insights/insights-runtime-extractor-7k285" Sep 10 13:18:16.595941 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:18:16.595832 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a5ede13b-c926-4b47-822c-384dc218e8f5-tmp-dir\") pod \"dns-default-9x5xs\" (UID: \"a5ede13b-c926-4b47-822c-384dc218e8f5\") " pod="openshift-dns/dns-default-9x5xs" Sep 10 13:18:16.595941 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:18:16.595861 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/2c8e9d2b-dd0d-4715-b367-54121669c328-cert\") pod \"ingress-canary-m7dz6\" (UID: \"2c8e9d2b-dd0d-4715-b367-54121669c328\") " pod="openshift-ingress-canary/ingress-canary-m7dz6" Sep 10 13:18:16.595941 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:18:16.595885 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-knqnp\" (UniqueName: \"kubernetes.io/projected/2c8e9d2b-dd0d-4715-b367-54121669c328-kube-api-access-knqnp\") pod \"ingress-canary-m7dz6\" (UID: \"2c8e9d2b-dd0d-4715-b367-54121669c328\") " pod="openshift-ingress-canary/ingress-canary-m7dz6" Sep 10 13:18:16.595941 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:18:16.595903 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/67184dde-9271-4b22-adb6-9a94adaf6c57-data-volume\") pod \"insights-runtime-extractor-7k285\" (UID: \"67184dde-9271-4b22-adb6-9a94adaf6c57\") " pod="openshift-insights/insights-runtime-extractor-7k285" Sep 10 13:18:16.696994 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:18:16.696953 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/a5ede13b-c926-4b47-822c-384dc218e8f5-config-volume\") pod \"dns-default-9x5xs\" (UID: \"a5ede13b-c926-4b47-822c-384dc218e8f5\") " pod="openshift-dns/dns-default-9x5xs" Sep 10 13:18:16.696994 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:18:16.696994 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zs6lv\" (UniqueName: \"kubernetes.io/projected/a5ede13b-c926-4b47-822c-384dc218e8f5-kube-api-access-zs6lv\") pod \"dns-default-9x5xs\" (UID: \"a5ede13b-c926-4b47-822c-384dc218e8f5\") " pod="openshift-dns/dns-default-9x5xs" Sep 10 13:18:16.697177 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:18:16.697017 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/67184dde-9271-4b22-adb6-9a94adaf6c57-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-7k285\" (UID: \"67184dde-9271-4b22-adb6-9a94adaf6c57\") " pod="openshift-insights/insights-runtime-extractor-7k285" Sep 10 13:18:16.697177 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:18:16.697045 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/a5ede13b-c926-4b47-822c-384dc218e8f5-metrics-tls\") pod \"dns-default-9x5xs\" (UID: \"a5ede13b-c926-4b47-822c-384dc218e8f5\") " pod="openshift-dns/dns-default-9x5xs" Sep 10 13:18:16.697177 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:18:16.697072 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/67184dde-9271-4b22-adb6-9a94adaf6c57-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-7k285\" (UID: \"67184dde-9271-4b22-adb6-9a94adaf6c57\") " pod="openshift-insights/insights-runtime-extractor-7k285" Sep 10 13:18:16.697177 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:18:16.697100 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/67184dde-9271-4b22-adb6-9a94adaf6c57-crio-socket\") pod \"insights-runtime-extractor-7k285\" (UID: \"67184dde-9271-4b22-adb6-9a94adaf6c57\") " pod="openshift-insights/insights-runtime-extractor-7k285" Sep 10 13:18:16.697177 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:18:16.697124 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5dfgh\" (UniqueName: \"kubernetes.io/projected/67184dde-9271-4b22-adb6-9a94adaf6c57-kube-api-access-5dfgh\") pod \"insights-runtime-extractor-7k285\" (UID: \"67184dde-9271-4b22-adb6-9a94adaf6c57\") " pod="openshift-insights/insights-runtime-extractor-7k285" Sep 10 13:18:16.697177 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:18:16.697155 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a5ede13b-c926-4b47-822c-384dc218e8f5-tmp-dir\") pod \"dns-default-9x5xs\" (UID: \"a5ede13b-c926-4b47-822c-384dc218e8f5\") " pod="openshift-dns/dns-default-9x5xs" Sep 10 13:18:16.697177 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:18:16.697175 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/2c8e9d2b-dd0d-4715-b367-54121669c328-cert\") pod \"ingress-canary-m7dz6\" (UID: \"2c8e9d2b-dd0d-4715-b367-54121669c328\") " pod="openshift-ingress-canary/ingress-canary-m7dz6" Sep 10 13:18:16.697531 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:18:16.697199 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-knqnp\" (UniqueName: \"kubernetes.io/projected/2c8e9d2b-dd0d-4715-b367-54121669c328-kube-api-access-knqnp\") pod \"ingress-canary-m7dz6\" (UID: \"2c8e9d2b-dd0d-4715-b367-54121669c328\") " pod="openshift-ingress-canary/ingress-canary-m7dz6" Sep 10 13:18:16.697531 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:18:16.697218 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/67184dde-9271-4b22-adb6-9a94adaf6c57-crio-socket\") pod \"insights-runtime-extractor-7k285\" (UID: \"67184dde-9271-4b22-adb6-9a94adaf6c57\") " pod="openshift-insights/insights-runtime-extractor-7k285" Sep 10 13:18:16.697531 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:18:16.697226 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/67184dde-9271-4b22-adb6-9a94adaf6c57-data-volume\") pod \"insights-runtime-extractor-7k285\" (UID: \"67184dde-9271-4b22-adb6-9a94adaf6c57\") " pod="openshift-insights/insights-runtime-extractor-7k285" Sep 10 13:18:16.697653 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:18:16.697583 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/67184dde-9271-4b22-adb6-9a94adaf6c57-data-volume\") pod \"insights-runtime-extractor-7k285\" (UID: \"67184dde-9271-4b22-adb6-9a94adaf6c57\") " pod="openshift-insights/insights-runtime-extractor-7k285" Sep 10 13:18:16.697704 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:18:16.697665 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a5ede13b-c926-4b47-822c-384dc218e8f5-tmp-dir\") pod \"dns-default-9x5xs\" (UID: \"a5ede13b-c926-4b47-822c-384dc218e8f5\") " pod="openshift-dns/dns-default-9x5xs" Sep 10 13:18:16.697755 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:18:16.697713 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/a5ede13b-c926-4b47-822c-384dc218e8f5-config-volume\") pod \"dns-default-9x5xs\" (UID: \"a5ede13b-c926-4b47-822c-384dc218e8f5\") " pod="openshift-dns/dns-default-9x5xs" Sep 10 13:18:16.697755 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:18:16.697716 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/67184dde-9271-4b22-adb6-9a94adaf6c57-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-7k285\" (UID: \"67184dde-9271-4b22-adb6-9a94adaf6c57\") " pod="openshift-insights/insights-runtime-extractor-7k285" Sep 10 13:18:16.701189 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:18:16.701161 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/67184dde-9271-4b22-adb6-9a94adaf6c57-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-7k285\" (UID: \"67184dde-9271-4b22-adb6-9a94adaf6c57\") " pod="openshift-insights/insights-runtime-extractor-7k285" Sep 10 13:18:16.701308 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:18:16.701219 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/a5ede13b-c926-4b47-822c-384dc218e8f5-metrics-tls\") pod \"dns-default-9x5xs\" (UID: \"a5ede13b-c926-4b47-822c-384dc218e8f5\") " pod="openshift-dns/dns-default-9x5xs" Sep 10 13:18:16.701308 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:18:16.701280 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/2c8e9d2b-dd0d-4715-b367-54121669c328-cert\") pod \"ingress-canary-m7dz6\" (UID: \"2c8e9d2b-dd0d-4715-b367-54121669c328\") " pod="openshift-ingress-canary/ingress-canary-m7dz6" Sep 10 13:18:16.712260 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:18:16.712203 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5dfgh\" (UniqueName: \"kubernetes.io/projected/67184dde-9271-4b22-adb6-9a94adaf6c57-kube-api-access-5dfgh\") pod \"insights-runtime-extractor-7k285\" (UID: \"67184dde-9271-4b22-adb6-9a94adaf6c57\") " pod="openshift-insights/insights-runtime-extractor-7k285" Sep 10 13:18:16.712455 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:18:16.712437 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zs6lv\" (UniqueName: \"kubernetes.io/projected/a5ede13b-c926-4b47-822c-384dc218e8f5-kube-api-access-zs6lv\") pod \"dns-default-9x5xs\" (UID: \"a5ede13b-c926-4b47-822c-384dc218e8f5\") " pod="openshift-dns/dns-default-9x5xs" Sep 10 13:18:16.712572 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:18:16.712553 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-knqnp\" (UniqueName: \"kubernetes.io/projected/2c8e9d2b-dd0d-4715-b367-54121669c328-kube-api-access-knqnp\") pod \"ingress-canary-m7dz6\" (UID: \"2c8e9d2b-dd0d-4715-b367-54121669c328\") " pod="openshift-ingress-canary/ingress-canary-m7dz6" Sep 10 13:18:16.818676 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:18:16.818642 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-9x5xs" Sep 10 13:18:16.823472 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:18:16.823458 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-m7dz6" Sep 10 13:18:16.840350 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:18:16.840321 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-7k285" Sep 10 13:18:17.120182 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:18:17.120113 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-m7dz6"] Sep 10 13:18:17.123142 ip-10-0-139-174 kubenswrapper[2557]: W0910 13:18:17.123117 2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2c8e9d2b_dd0d_4715_b367_54121669c328.slice/crio-88599bb0723f0e47b1970867d9fedd2c71253cd1abcf9e1bbaf9a9b1afb5bded WatchSource:0}: Error finding container 88599bb0723f0e47b1970867d9fedd2c71253cd1abcf9e1bbaf9a9b1afb5bded: Status 404 returned error can't find the container with id 88599bb0723f0e47b1970867d9fedd2c71253cd1abcf9e1bbaf9a9b1afb5bded Sep 10 13:18:17.125158 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:18:17.125136 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-9x5xs"] Sep 10 13:18:17.127584 ip-10-0-139-174 kubenswrapper[2557]: W0910 13:18:17.127559 2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda5ede13b_c926_4b47_822c_384dc218e8f5.slice/crio-6fef4434db487bfcf23d569151e31b380d1448ce33dc32d04d355b273245c274 WatchSource:0}: Error finding container 6fef4434db487bfcf23d569151e31b380d1448ce33dc32d04d355b273245c274: Status 404 returned error can't find the container with id 6fef4434db487bfcf23d569151e31b380d1448ce33dc32d04d355b273245c274 Sep 10 13:18:17.141530 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:18:17.141507 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-7k285"] Sep 10 13:18:17.144742 ip-10-0-139-174 kubenswrapper[2557]: W0910 13:18:17.144723 2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod67184dde_9271_4b22_adb6_9a94adaf6c57.slice/crio-3e26bc314261bbcc74610465b22b2003118e76b6e872fe2b428e0978eb4c15e2 WatchSource:0}: Error finding container 3e26bc314261bbcc74610465b22b2003118e76b6e872fe2b428e0978eb4c15e2: Status 404 returned error can't find the container with id 3e26bc314261bbcc74610465b22b2003118e76b6e872fe2b428e0978eb4c15e2 Sep 10 13:18:17.507738 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:18:17.507702 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-7k285" event={"ID":"67184dde-9271-4b22-adb6-9a94adaf6c57","Type":"ContainerStarted","Data":"a82c521f18e27a1e0861c6ac6d2f5b0e6af43a074e1271a599a002eb893c5654"} Sep 10 13:18:17.508116 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:18:17.507804 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-7k285" event={"ID":"67184dde-9271-4b22-adb6-9a94adaf6c57","Type":"ContainerStarted","Data":"3e26bc314261bbcc74610465b22b2003118e76b6e872fe2b428e0978eb4c15e2"} Sep 10 13:18:17.509002 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:18:17.508970 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-m7dz6" event={"ID":"2c8e9d2b-dd0d-4715-b367-54121669c328","Type":"ContainerStarted","Data":"88599bb0723f0e47b1970867d9fedd2c71253cd1abcf9e1bbaf9a9b1afb5bded"} Sep 10 13:18:17.510199 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:18:17.510171 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-9x5xs" event={"ID":"a5ede13b-c926-4b47-822c-384dc218e8f5","Type":"ContainerStarted","Data":"6fef4434db487bfcf23d569151e31b380d1448ce33dc32d04d355b273245c274"} Sep 10 13:18:17.811448 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:18:17.811346 2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-7448d46fdc-jj7z4"] Sep 10 13:18:17.813296 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:18:17.813273 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7448d46fdc-jj7z4" Sep 10 13:18:17.817293 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:18:17.817273 2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\"" Sep 10 13:18:17.818869 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:18:17.818338 2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\"" Sep 10 13:18:17.820457 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:18:17.819854 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-vhx9c\"" Sep 10 13:18:17.820457 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:18:17.820273 2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\"" Sep 10 13:18:17.820688 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:18:17.820656 2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\"" Sep 10 13:18:17.820787 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:18:17.820740 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\"" Sep 10 13:18:17.820860 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:18:17.820806 2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\"" Sep 10 13:18:17.820896 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:18:17.820876 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\"" Sep 10 13:18:17.828778 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:18:17.828754 2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"trusted-ca-bundle\"" Sep 10 13:18:17.828885 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:18:17.828758 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-7448d46fdc-jj7z4"] Sep 10 13:18:17.905114 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:18:17.905077 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/23407ee4-a22b-4bde-81f0-899eb8532f49-console-config\") pod \"console-7448d46fdc-jj7z4\" (UID: \"23407ee4-a22b-4bde-81f0-899eb8532f49\") " pod="openshift-console/console-7448d46fdc-jj7z4" Sep 10 13:18:17.905297 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:18:17.905129 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/23407ee4-a22b-4bde-81f0-899eb8532f49-console-serving-cert\") pod \"console-7448d46fdc-jj7z4\" (UID: \"23407ee4-a22b-4bde-81f0-899eb8532f49\") " pod="openshift-console/console-7448d46fdc-jj7z4" Sep 10 13:18:17.905297 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:18:17.905162 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/23407ee4-a22b-4bde-81f0-899eb8532f49-service-ca\") pod \"console-7448d46fdc-jj7z4\" (UID: \"23407ee4-a22b-4bde-81f0-899eb8532f49\") " pod="openshift-console/console-7448d46fdc-jj7z4" Sep 10 13:18:17.905297 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:18:17.905188 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/23407ee4-a22b-4bde-81f0-899eb8532f49-oauth-serving-cert\") pod \"console-7448d46fdc-jj7z4\" (UID: \"23407ee4-a22b-4bde-81f0-899eb8532f49\") " pod="openshift-console/console-7448d46fdc-jj7z4" Sep 10 13:18:17.905297 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:18:17.905241 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/23407ee4-a22b-4bde-81f0-899eb8532f49-console-oauth-config\") pod \"console-7448d46fdc-jj7z4\" (UID: \"23407ee4-a22b-4bde-81f0-899eb8532f49\") " pod="openshift-console/console-7448d46fdc-jj7z4" Sep 10 13:18:17.905546 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:18:17.905301 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/23407ee4-a22b-4bde-81f0-899eb8532f49-trusted-ca-bundle\") pod \"console-7448d46fdc-jj7z4\" (UID: \"23407ee4-a22b-4bde-81f0-899eb8532f49\") " pod="openshift-console/console-7448d46fdc-jj7z4" Sep 10 13:18:17.905546 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:18:17.905326 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-75ww7\" (UniqueName: \"kubernetes.io/projected/23407ee4-a22b-4bde-81f0-899eb8532f49-kube-api-access-75ww7\") pod \"console-7448d46fdc-jj7z4\" (UID: \"23407ee4-a22b-4bde-81f0-899eb8532f49\") " pod="openshift-console/console-7448d46fdc-jj7z4" Sep 10 13:18:18.006201 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:18:18.006174 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/23407ee4-a22b-4bde-81f0-899eb8532f49-console-config\") pod \"console-7448d46fdc-jj7z4\" (UID: \"23407ee4-a22b-4bde-81f0-899eb8532f49\") " pod="openshift-console/console-7448d46fdc-jj7z4" Sep 10 13:18:18.006304 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:18:18.006220 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/23407ee4-a22b-4bde-81f0-899eb8532f49-console-serving-cert\") pod \"console-7448d46fdc-jj7z4\" (UID: \"23407ee4-a22b-4bde-81f0-899eb8532f49\") " pod="openshift-console/console-7448d46fdc-jj7z4" Sep 10 13:18:18.006304 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:18:18.006250 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/23407ee4-a22b-4bde-81f0-899eb8532f49-service-ca\") pod \"console-7448d46fdc-jj7z4\" (UID: \"23407ee4-a22b-4bde-81f0-899eb8532f49\") " pod="openshift-console/console-7448d46fdc-jj7z4" Sep 10 13:18:18.006304 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:18:18.006275 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/23407ee4-a22b-4bde-81f0-899eb8532f49-oauth-serving-cert\") pod \"console-7448d46fdc-jj7z4\" (UID: \"23407ee4-a22b-4bde-81f0-899eb8532f49\") " pod="openshift-console/console-7448d46fdc-jj7z4" Sep 10 13:18:18.006483 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:18:18.006308 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/23407ee4-a22b-4bde-81f0-899eb8532f49-console-oauth-config\") pod \"console-7448d46fdc-jj7z4\" (UID: \"23407ee4-a22b-4bde-81f0-899eb8532f49\") " pod="openshift-console/console-7448d46fdc-jj7z4" Sep 10 13:18:18.006483 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:18:18.006357 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/23407ee4-a22b-4bde-81f0-899eb8532f49-trusted-ca-bundle\") pod \"console-7448d46fdc-jj7z4\" (UID: \"23407ee4-a22b-4bde-81f0-899eb8532f49\") " pod="openshift-console/console-7448d46fdc-jj7z4" Sep 10 13:18:18.006483 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:18:18.006381 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-75ww7\" (UniqueName: \"kubernetes.io/projected/23407ee4-a22b-4bde-81f0-899eb8532f49-kube-api-access-75ww7\") pod \"console-7448d46fdc-jj7z4\" (UID: \"23407ee4-a22b-4bde-81f0-899eb8532f49\") " pod="openshift-console/console-7448d46fdc-jj7z4" Sep 10 13:18:18.007214 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:18:18.006898 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/23407ee4-a22b-4bde-81f0-899eb8532f49-console-config\") pod \"console-7448d46fdc-jj7z4\" (UID: \"23407ee4-a22b-4bde-81f0-899eb8532f49\") " pod="openshift-console/console-7448d46fdc-jj7z4" Sep 10 13:18:18.007214 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:18:18.006977 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/23407ee4-a22b-4bde-81f0-899eb8532f49-service-ca\") pod \"console-7448d46fdc-jj7z4\" (UID: \"23407ee4-a22b-4bde-81f0-899eb8532f49\") " pod="openshift-console/console-7448d46fdc-jj7z4" Sep 10 13:18:18.007354 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:18:18.007301 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/23407ee4-a22b-4bde-81f0-899eb8532f49-oauth-serving-cert\") pod \"console-7448d46fdc-jj7z4\" (UID: \"23407ee4-a22b-4bde-81f0-899eb8532f49\") " pod="openshift-console/console-7448d46fdc-jj7z4" Sep 10 13:18:18.008088 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:18:18.008063 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/23407ee4-a22b-4bde-81f0-899eb8532f49-trusted-ca-bundle\") pod \"console-7448d46fdc-jj7z4\" (UID: \"23407ee4-a22b-4bde-81f0-899eb8532f49\") " pod="openshift-console/console-7448d46fdc-jj7z4" Sep 10 13:18:18.009764 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:18:18.009718 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/23407ee4-a22b-4bde-81f0-899eb8532f49-console-oauth-config\") pod \"console-7448d46fdc-jj7z4\" (UID: \"23407ee4-a22b-4bde-81f0-899eb8532f49\") " pod="openshift-console/console-7448d46fdc-jj7z4" Sep 10 13:18:18.009924 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:18:18.009902 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/23407ee4-a22b-4bde-81f0-899eb8532f49-console-serving-cert\") pod \"console-7448d46fdc-jj7z4\" (UID: \"23407ee4-a22b-4bde-81f0-899eb8532f49\") " pod="openshift-console/console-7448d46fdc-jj7z4" Sep 10 13:18:18.015640 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:18:18.015588 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-75ww7\" (UniqueName: \"kubernetes.io/projected/23407ee4-a22b-4bde-81f0-899eb8532f49-kube-api-access-75ww7\") pod \"console-7448d46fdc-jj7z4\" (UID: \"23407ee4-a22b-4bde-81f0-899eb8532f49\") " pod="openshift-console/console-7448d46fdc-jj7z4" Sep 10 13:18:18.128143 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:18:18.128065 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7448d46fdc-jj7z4" Sep 10 13:18:18.214349 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:18:18.214321 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-6ggtx" Sep 10 13:18:18.215032 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:18:18.214954 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-7vvkj" Sep 10 13:18:18.215154 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:18:18.215091 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-phs5r" Sep 10 13:18:18.218989 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:18:18.218967 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-w47rx\"" Sep 10 13:18:18.220913 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:18:18.220785 2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Sep 10 13:18:18.220913 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:18:18.220788 2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Sep 10 13:18:18.221385 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:18:18.221365 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Sep 10 13:18:18.221494 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:18:18.221461 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-5vlq4\"" Sep 10 13:18:18.221669 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:18:18.221654 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Sep 10 13:18:18.409722 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:18:18.409647 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-7448d46fdc-jj7z4"] Sep 10 13:18:18.516702 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:18:18.516660 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-7k285" event={"ID":"67184dde-9271-4b22-adb6-9a94adaf6c57","Type":"ContainerStarted","Data":"195750ce0557b8675849bb35cfe1495240fdf32dc4fed1f2187fb160d0403b71"} Sep 10 13:18:18.716998 ip-10-0-139-174 kubenswrapper[2557]: W0910 13:18:18.716965 2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod23407ee4_a22b_4bde_81f0_899eb8532f49.slice/crio-95e49e1bff384635b55269c587cfa81c0da8ef827ba983f421270434a75eb072 WatchSource:0}: Error finding container 95e49e1bff384635b55269c587cfa81c0da8ef827ba983f421270434a75eb072: Status 404 returned error can't find the container with id 95e49e1bff384635b55269c587cfa81c0da8ef827ba983f421270434a75eb072 Sep 10 13:18:19.521624 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:18:19.521582 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-9x5xs" event={"ID":"a5ede13b-c926-4b47-822c-384dc218e8f5","Type":"ContainerStarted","Data":"f9788c0341e958f7f5229fdcd03abcea2d7cb0fb4be487c15aa1c646ec573e02"} Sep 10 13:18:19.523061 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:18:19.523037 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7448d46fdc-jj7z4" event={"ID":"23407ee4-a22b-4bde-81f0-899eb8532f49","Type":"ContainerStarted","Data":"95e49e1bff384635b55269c587cfa81c0da8ef827ba983f421270434a75eb072"} Sep 10 13:18:20.529549 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:18:20.529510 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-7k285" event={"ID":"67184dde-9271-4b22-adb6-9a94adaf6c57","Type":"ContainerStarted","Data":"098d8056d874aa5e9dbe67720f7b8603b8ffcb3f9a6d99f2fdf9d971acb5eb3c"} Sep 10 13:18:20.531247 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:18:20.531219 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-m7dz6" event={"ID":"2c8e9d2b-dd0d-4715-b367-54121669c328","Type":"ContainerStarted","Data":"c888a2cc2f1162b0edc9079d1a6fec54b77efac666a6b665890916a62669a277"} Sep 10 13:18:20.533768 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:18:20.533716 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-9x5xs" event={"ID":"a5ede13b-c926-4b47-822c-384dc218e8f5","Type":"ContainerStarted","Data":"51ec2f402de8c55441f811cac78531911a324bb3f20ba7a53179588d5aa03355"} Sep 10 13:18:20.533923 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:18:20.533891 2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-9x5xs" Sep 10 13:18:20.548163 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:18:20.548118 2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-7k285" podStartSLOduration=1.8053278449999999 podStartE2EDuration="4.548104292s" totalImagesPullingTime="2.742776447s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-10 13:18:16 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 13:18:20.547472815 +0000 UTC m=+60.854467228" watchObservedRunningTime="2026-09-10 13:18:20.548104292 +0000 UTC m=+60.855098705" Sep 10 13:18:20.579624 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:18:20.579565 2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-9x5xs" podStartSLOduration=2.961683758 podStartE2EDuration="4.579549866s" totalImagesPullingTime="1.617866108s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 13:18:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 13:18:20.577854627 +0000 UTC m=+60.884849039" watchObservedRunningTime="2026-09-10 13:18:20.579549866 +0000 UTC m=+60.886544280" Sep 10 13:18:20.612160 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:18:20.612112 2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-m7dz6" podStartSLOduration=1.7078860759999999 podStartE2EDuration="4.612098188s" totalImagesPullingTime="2.904212112s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 13:18:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 13:18:20.610844903 +0000 UTC m=+60.917839315" watchObservedRunningTime="2026-09-10 13:18:20.612098188 +0000 UTC m=+60.919092600" Sep 10 13:18:20.827725 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:18:20.827637 2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/openshift-state-metrics-657fd6c54-vnznb"] Sep 10 13:18:20.829983 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:18:20.829960 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-657fd6c54-vnznb" Sep 10 13:18:20.831873 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:18:20.831852 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-tls\"" Sep 10 13:18:20.831994 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:18:20.831857 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-kube-rbac-proxy-config\"" Sep 10 13:18:20.832644 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:18:20.832625 2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\"" Sep 10 13:18:20.832758 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:18:20.832682 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-dockercfg-ndws5\"" Sep 10 13:18:20.832820 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:18:20.832771 2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\"" Sep 10 13:18:20.832958 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:18:20.832937 2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\"" Sep 10 13:18:20.834584 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:18:20.834566 2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/kube-state-metrics-74cb57df7-dtddc"] Sep 10 13:18:20.836752 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:18:20.836736 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-74cb57df7-dtddc" Sep 10 13:18:20.838664 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:18:20.838648 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-kube-rbac-proxy-config\"" Sep 10 13:18:20.838757 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:18:20.838719 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-tls\"" Sep 10 13:18:20.838808 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:18:20.838768 2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-custom-resource-state-configmap\"" Sep 10 13:18:20.838870 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:18:20.838857 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-dockercfg-mbp8t\"" Sep 10 13:18:20.843406 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:18:20.843385 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-657fd6c54-vnznb"] Sep 10 13:18:20.850520 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:18:20.850499 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-74cb57df7-dtddc"] Sep 10 13:18:20.870990 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:18:20.870961 2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-pvp5f"] Sep 10 13:18:20.873240 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:18:20.873218 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-pvp5f" Sep 10 13:18:20.875321 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:18:20.875296 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\"" Sep 10 13:18:20.875774 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:18:20.875759 2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\"" Sep 10 13:18:20.876628 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:18:20.876602 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\"" Sep 10 13:18:20.876989 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:18:20.876971 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-l4r8x\"" Sep 10 13:18:20.928594 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:18:20.928567 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8xqnd\" (UniqueName: \"kubernetes.io/projected/f42e5bbe-1bd2-472a-b55d-786b3a65fba9-kube-api-access-8xqnd\") pod \"openshift-state-metrics-657fd6c54-vnznb\" (UID: \"f42e5bbe-1bd2-472a-b55d-786b3a65fba9\") " pod="openshift-monitoring/openshift-state-metrics-657fd6c54-vnznb" Sep 10 13:18:20.928725 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:18:20.928611 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/f42e5bbe-1bd2-472a-b55d-786b3a65fba9-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-657fd6c54-vnznb\" (UID: \"f42e5bbe-1bd2-472a-b55d-786b3a65fba9\") " pod="openshift-monitoring/openshift-state-metrics-657fd6c54-vnznb" Sep 10 13:18:20.928725 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:18:20.928644 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/9eea609d-ef83-4cdd-aacb-d0719e24b978-volume-directive-shadow\") pod \"kube-state-metrics-74cb57df7-dtddc\" (UID: \"9eea609d-ef83-4cdd-aacb-d0719e24b978\") " pod="openshift-monitoring/kube-state-metrics-74cb57df7-dtddc" Sep 10 13:18:20.928725 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:18:20.928671 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/fee71c0c-b8c8-4802-b2dd-e8c2ba7389cc-node-exporter-textfile\") pod \"node-exporter-pvp5f\" (UID: \"fee71c0c-b8c8-4802-b2dd-e8c2ba7389cc\") " pod="openshift-monitoring/node-exporter-pvp5f" Sep 10 13:18:20.928725 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:18:20.928707 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/9eea609d-ef83-4cdd-aacb-d0719e24b978-kube-state-metrics-tls\") pod \"kube-state-metrics-74cb57df7-dtddc\" (UID: \"9eea609d-ef83-4cdd-aacb-d0719e24b978\") " pod="openshift-monitoring/kube-state-metrics-74cb57df7-dtddc" Sep 10 13:18:20.928725 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:18:20.928721 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rn7xz\" (UniqueName: \"kubernetes.io/projected/9eea609d-ef83-4cdd-aacb-d0719e24b978-kube-api-access-rn7xz\") pod \"kube-state-metrics-74cb57df7-dtddc\" (UID: \"9eea609d-ef83-4cdd-aacb-d0719e24b978\") " pod="openshift-monitoring/kube-state-metrics-74cb57df7-dtddc" Sep 10 13:18:20.929016 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:18:20.928737 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/f42e5bbe-1bd2-472a-b55d-786b3a65fba9-metrics-client-ca\") pod \"openshift-state-metrics-657fd6c54-vnznb\" (UID: \"f42e5bbe-1bd2-472a-b55d-786b3a65fba9\") " pod="openshift-monitoring/openshift-state-metrics-657fd6c54-vnznb" Sep 10 13:18:20.929016 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:18:20.928755 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/fee71c0c-b8c8-4802-b2dd-e8c2ba7389cc-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-pvp5f\" (UID: \"fee71c0c-b8c8-4802-b2dd-e8c2ba7389cc\") " pod="openshift-monitoring/node-exporter-pvp5f" Sep 10 13:18:20.929016 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:18:20.928801 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/fee71c0c-b8c8-4802-b2dd-e8c2ba7389cc-node-exporter-tls\") pod \"node-exporter-pvp5f\" (UID: \"fee71c0c-b8c8-4802-b2dd-e8c2ba7389cc\") " pod="openshift-monitoring/node-exporter-pvp5f" Sep 10 13:18:20.929016 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:18:20.928936 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/9eea609d-ef83-4cdd-aacb-d0719e24b978-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-74cb57df7-dtddc\" (UID: \"9eea609d-ef83-4cdd-aacb-d0719e24b978\") " pod="openshift-monitoring/kube-state-metrics-74cb57df7-dtddc" Sep 10 13:18:20.929016 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:18:20.928981 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/fee71c0c-b8c8-4802-b2dd-e8c2ba7389cc-sys\") pod \"node-exporter-pvp5f\" (UID: \"fee71c0c-b8c8-4802-b2dd-e8c2ba7389cc\") " pod="openshift-monitoring/node-exporter-pvp5f" Sep 10 13:18:20.929016 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:18:20.929001 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kmn7t\" (UniqueName: \"kubernetes.io/projected/fee71c0c-b8c8-4802-b2dd-e8c2ba7389cc-kube-api-access-kmn7t\") pod \"node-exporter-pvp5f\" (UID: \"fee71c0c-b8c8-4802-b2dd-e8c2ba7389cc\") " pod="openshift-monitoring/node-exporter-pvp5f" Sep 10 13:18:20.929351 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:18:20.929101 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/fee71c0c-b8c8-4802-b2dd-e8c2ba7389cc-root\") pod \"node-exporter-pvp5f\" (UID: \"fee71c0c-b8c8-4802-b2dd-e8c2ba7389cc\") " pod="openshift-monitoring/node-exporter-pvp5f" Sep 10 13:18:20.929351 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:18:20.929120 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/fee71c0c-b8c8-4802-b2dd-e8c2ba7389cc-metrics-client-ca\") pod \"node-exporter-pvp5f\" (UID: \"fee71c0c-b8c8-4802-b2dd-e8c2ba7389cc\") " pod="openshift-monitoring/node-exporter-pvp5f" Sep 10 13:18:20.929351 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:18:20.929147 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/f42e5bbe-1bd2-472a-b55d-786b3a65fba9-openshift-state-metrics-tls\") pod \"openshift-state-metrics-657fd6c54-vnznb\" (UID: \"f42e5bbe-1bd2-472a-b55d-786b3a65fba9\") " pod="openshift-monitoring/openshift-state-metrics-657fd6c54-vnznb" Sep 10 13:18:20.929351 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:18:20.929171 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/fee71c0c-b8c8-4802-b2dd-e8c2ba7389cc-node-exporter-wtmp\") pod \"node-exporter-pvp5f\" (UID: \"fee71c0c-b8c8-4802-b2dd-e8c2ba7389cc\") " pod="openshift-monitoring/node-exporter-pvp5f" Sep 10 13:18:20.929351 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:18:20.929199 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/fee71c0c-b8c8-4802-b2dd-e8c2ba7389cc-node-exporter-accelerators-collector-config\") pod \"node-exporter-pvp5f\" (UID: \"fee71c0c-b8c8-4802-b2dd-e8c2ba7389cc\") " pod="openshift-monitoring/node-exporter-pvp5f" Sep 10 13:18:20.929351 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:18:20.929255 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/9eea609d-ef83-4cdd-aacb-d0719e24b978-metrics-client-ca\") pod \"kube-state-metrics-74cb57df7-dtddc\" (UID: \"9eea609d-ef83-4cdd-aacb-d0719e24b978\") " pod="openshift-monitoring/kube-state-metrics-74cb57df7-dtddc" Sep 10 13:18:20.929351 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:18:20.929285 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/9eea609d-ef83-4cdd-aacb-d0719e24b978-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-74cb57df7-dtddc\" (UID: \"9eea609d-ef83-4cdd-aacb-d0719e24b978\") " pod="openshift-monitoring/kube-state-metrics-74cb57df7-dtddc" Sep 10 13:18:21.030052 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:18:21.030007 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/9eea609d-ef83-4cdd-aacb-d0719e24b978-volume-directive-shadow\") pod \"kube-state-metrics-74cb57df7-dtddc\" (UID: \"9eea609d-ef83-4cdd-aacb-d0719e24b978\") " pod="openshift-monitoring/kube-state-metrics-74cb57df7-dtddc" Sep 10 13:18:21.030052 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:18:21.030050 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/fee71c0c-b8c8-4802-b2dd-e8c2ba7389cc-node-exporter-textfile\") pod \"node-exporter-pvp5f\" (UID: \"fee71c0c-b8c8-4802-b2dd-e8c2ba7389cc\") " pod="openshift-monitoring/node-exporter-pvp5f" Sep 10 13:18:21.030276 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:18:21.030090 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/9eea609d-ef83-4cdd-aacb-d0719e24b978-kube-state-metrics-tls\") pod \"kube-state-metrics-74cb57df7-dtddc\" (UID: \"9eea609d-ef83-4cdd-aacb-d0719e24b978\") " pod="openshift-monitoring/kube-state-metrics-74cb57df7-dtddc" Sep 10 13:18:21.030276 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:18:21.030108 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rn7xz\" (UniqueName: \"kubernetes.io/projected/9eea609d-ef83-4cdd-aacb-d0719e24b978-kube-api-access-rn7xz\") pod \"kube-state-metrics-74cb57df7-dtddc\" (UID: \"9eea609d-ef83-4cdd-aacb-d0719e24b978\") " pod="openshift-monitoring/kube-state-metrics-74cb57df7-dtddc" Sep 10 13:18:21.030276 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:18:21.030124 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/f42e5bbe-1bd2-472a-b55d-786b3a65fba9-metrics-client-ca\") pod \"openshift-state-metrics-657fd6c54-vnznb\" (UID: \"f42e5bbe-1bd2-472a-b55d-786b3a65fba9\") " pod="openshift-monitoring/openshift-state-metrics-657fd6c54-vnznb" Sep 10 13:18:21.030276 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:18:21.030143 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/fee71c0c-b8c8-4802-b2dd-e8c2ba7389cc-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-pvp5f\" (UID: \"fee71c0c-b8c8-4802-b2dd-e8c2ba7389cc\") " pod="openshift-monitoring/node-exporter-pvp5f" Sep 10 13:18:21.030276 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:18:21.030170 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/fee71c0c-b8c8-4802-b2dd-e8c2ba7389cc-node-exporter-tls\") pod \"node-exporter-pvp5f\" (UID: \"fee71c0c-b8c8-4802-b2dd-e8c2ba7389cc\") " pod="openshift-monitoring/node-exporter-pvp5f" Sep 10 13:18:21.030276 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:18:21.030200 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/9eea609d-ef83-4cdd-aacb-d0719e24b978-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-74cb57df7-dtddc\" (UID: \"9eea609d-ef83-4cdd-aacb-d0719e24b978\") " pod="openshift-monitoring/kube-state-metrics-74cb57df7-dtddc" Sep 10 13:18:21.030276 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:18:21.030227 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/fee71c0c-b8c8-4802-b2dd-e8c2ba7389cc-sys\") pod \"node-exporter-pvp5f\" (UID: \"fee71c0c-b8c8-4802-b2dd-e8c2ba7389cc\") " pod="openshift-monitoring/node-exporter-pvp5f" Sep 10 13:18:21.030276 ip-10-0-139-174 kubenswrapper[2557]: E0910 13:18:21.030237 2557 secret.go:189] Couldn't get secret openshift-monitoring/kube-state-metrics-tls: secret "kube-state-metrics-tls" not found Sep 10 13:18:21.030276 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:18:21.030249 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kmn7t\" (UniqueName: \"kubernetes.io/projected/fee71c0c-b8c8-4802-b2dd-e8c2ba7389cc-kube-api-access-kmn7t\") pod \"node-exporter-pvp5f\" (UID: \"fee71c0c-b8c8-4802-b2dd-e8c2ba7389cc\") " pod="openshift-monitoring/node-exporter-pvp5f" Sep 10 13:18:21.030620 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:18:21.030280 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/fee71c0c-b8c8-4802-b2dd-e8c2ba7389cc-root\") pod \"node-exporter-pvp5f\" (UID: \"fee71c0c-b8c8-4802-b2dd-e8c2ba7389cc\") " pod="openshift-monitoring/node-exporter-pvp5f" Sep 10 13:18:21.030620 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:18:21.030307 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/fee71c0c-b8c8-4802-b2dd-e8c2ba7389cc-metrics-client-ca\") pod \"node-exporter-pvp5f\" (UID: \"fee71c0c-b8c8-4802-b2dd-e8c2ba7389cc\") " pod="openshift-monitoring/node-exporter-pvp5f" Sep 10 13:18:21.030620 ip-10-0-139-174 kubenswrapper[2557]: E0910 13:18:21.030324 2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9eea609d-ef83-4cdd-aacb-d0719e24b978-kube-state-metrics-tls podName:9eea609d-ef83-4cdd-aacb-d0719e24b978 nodeName:}" failed. No retries permitted until 2026-09-10 13:18:21.530302825 +0000 UTC m=+61.837297217 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-state-metrics-tls" (UniqueName: "kubernetes.io/secret/9eea609d-ef83-4cdd-aacb-d0719e24b978-kube-state-metrics-tls") pod "kube-state-metrics-74cb57df7-dtddc" (UID: "9eea609d-ef83-4cdd-aacb-d0719e24b978") : secret "kube-state-metrics-tls" not found Sep 10 13:18:21.030620 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:18:21.030353 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/f42e5bbe-1bd2-472a-b55d-786b3a65fba9-openshift-state-metrics-tls\") pod \"openshift-state-metrics-657fd6c54-vnznb\" (UID: \"f42e5bbe-1bd2-472a-b55d-786b3a65fba9\") " pod="openshift-monitoring/openshift-state-metrics-657fd6c54-vnznb" Sep 10 13:18:21.030620 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:18:21.030381 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/fee71c0c-b8c8-4802-b2dd-e8c2ba7389cc-node-exporter-wtmp\") pod \"node-exporter-pvp5f\" (UID: \"fee71c0c-b8c8-4802-b2dd-e8c2ba7389cc\") " pod="openshift-monitoring/node-exporter-pvp5f" Sep 10 13:18:21.030620 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:18:21.030392 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/9eea609d-ef83-4cdd-aacb-d0719e24b978-volume-directive-shadow\") pod \"kube-state-metrics-74cb57df7-dtddc\" (UID: \"9eea609d-ef83-4cdd-aacb-d0719e24b978\") " pod="openshift-monitoring/kube-state-metrics-74cb57df7-dtddc" Sep 10 13:18:21.030620 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:18:21.030436 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/fee71c0c-b8c8-4802-b2dd-e8c2ba7389cc-node-exporter-accelerators-collector-config\") pod \"node-exporter-pvp5f\" (UID: \"fee71c0c-b8c8-4802-b2dd-e8c2ba7389cc\") " pod="openshift-monitoring/node-exporter-pvp5f" Sep 10 13:18:21.030620 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:18:21.030488 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/9eea609d-ef83-4cdd-aacb-d0719e24b978-metrics-client-ca\") pod \"kube-state-metrics-74cb57df7-dtddc\" (UID: \"9eea609d-ef83-4cdd-aacb-d0719e24b978\") " pod="openshift-monitoring/kube-state-metrics-74cb57df7-dtddc" Sep 10 13:18:21.030620 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:18:21.030512 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/9eea609d-ef83-4cdd-aacb-d0719e24b978-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-74cb57df7-dtddc\" (UID: \"9eea609d-ef83-4cdd-aacb-d0719e24b978\") " pod="openshift-monitoring/kube-state-metrics-74cb57df7-dtddc" Sep 10 13:18:21.030620 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:18:21.030529 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8xqnd\" (UniqueName: \"kubernetes.io/projected/f42e5bbe-1bd2-472a-b55d-786b3a65fba9-kube-api-access-8xqnd\") pod \"openshift-state-metrics-657fd6c54-vnznb\" (UID: \"f42e5bbe-1bd2-472a-b55d-786b3a65fba9\") " pod="openshift-monitoring/openshift-state-metrics-657fd6c54-vnznb" Sep 10 13:18:21.030620 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:18:21.030547 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/f42e5bbe-1bd2-472a-b55d-786b3a65fba9-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-657fd6c54-vnznb\" (UID: \"f42e5bbe-1bd2-472a-b55d-786b3a65fba9\") " pod="openshift-monitoring/openshift-state-metrics-657fd6c54-vnznb" Sep 10 13:18:21.030620 ip-10-0-139-174 kubenswrapper[2557]: E0910 13:18:21.030561 2557 secret.go:189] Couldn't get secret openshift-monitoring/openshift-state-metrics-tls: secret "openshift-state-metrics-tls" not found Sep 10 13:18:21.030620 ip-10-0-139-174 kubenswrapper[2557]: E0910 13:18:21.030618 2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f42e5bbe-1bd2-472a-b55d-786b3a65fba9-openshift-state-metrics-tls podName:f42e5bbe-1bd2-472a-b55d-786b3a65fba9 nodeName:}" failed. No retries permitted until 2026-09-10 13:18:21.530600135 +0000 UTC m=+61.837594529 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "openshift-state-metrics-tls" (UniqueName: "kubernetes.io/secret/f42e5bbe-1bd2-472a-b55d-786b3a65fba9-openshift-state-metrics-tls") pod "openshift-state-metrics-657fd6c54-vnznb" (UID: "f42e5bbe-1bd2-472a-b55d-786b3a65fba9") : secret "openshift-state-metrics-tls" not found Sep 10 13:18:21.031078 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:18:21.030850 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/fee71c0c-b8c8-4802-b2dd-e8c2ba7389cc-metrics-client-ca\") pod \"node-exporter-pvp5f\" (UID: \"fee71c0c-b8c8-4802-b2dd-e8c2ba7389cc\") " pod="openshift-monitoring/node-exporter-pvp5f" Sep 10 13:18:21.031078 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:18:21.030907 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/fee71c0c-b8c8-4802-b2dd-e8c2ba7389cc-sys\") pod \"node-exporter-pvp5f\" (UID: \"fee71c0c-b8c8-4802-b2dd-e8c2ba7389cc\") " pod="openshift-monitoring/node-exporter-pvp5f" Sep 10 13:18:21.031186 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:18:21.031172 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/fee71c0c-b8c8-4802-b2dd-e8c2ba7389cc-root\") pod \"node-exporter-pvp5f\" (UID: \"fee71c0c-b8c8-4802-b2dd-e8c2ba7389cc\") " pod="openshift-monitoring/node-exporter-pvp5f" Sep 10 13:18:21.031447 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:18:21.030487 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/fee71c0c-b8c8-4802-b2dd-e8c2ba7389cc-node-exporter-textfile\") pod \"node-exporter-pvp5f\" (UID: \"fee71c0c-b8c8-4802-b2dd-e8c2ba7389cc\") " pod="openshift-monitoring/node-exporter-pvp5f" Sep 10 13:18:21.031447 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:18:21.031311 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/f42e5bbe-1bd2-472a-b55d-786b3a65fba9-metrics-client-ca\") pod \"openshift-state-metrics-657fd6c54-vnznb\" (UID: \"f42e5bbe-1bd2-472a-b55d-786b3a65fba9\") " pod="openshift-monitoring/openshift-state-metrics-657fd6c54-vnznb" Sep 10 13:18:21.031447 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:18:21.031311 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/9eea609d-ef83-4cdd-aacb-d0719e24b978-metrics-client-ca\") pod \"kube-state-metrics-74cb57df7-dtddc\" (UID: \"9eea609d-ef83-4cdd-aacb-d0719e24b978\") " pod="openshift-monitoring/kube-state-metrics-74cb57df7-dtddc" Sep 10 13:18:21.031447 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:18:21.031439 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/fee71c0c-b8c8-4802-b2dd-e8c2ba7389cc-node-exporter-wtmp\") pod \"node-exporter-pvp5f\" (UID: \"fee71c0c-b8c8-4802-b2dd-e8c2ba7389cc\") " pod="openshift-monitoring/node-exporter-pvp5f" Sep 10 13:18:21.031745 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:18:21.031501 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/9eea609d-ef83-4cdd-aacb-d0719e24b978-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-74cb57df7-dtddc\" (UID: \"9eea609d-ef83-4cdd-aacb-d0719e24b978\") " pod="openshift-monitoring/kube-state-metrics-74cb57df7-dtddc" Sep 10 13:18:21.032537 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:18:21.032509 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/fee71c0c-b8c8-4802-b2dd-e8c2ba7389cc-node-exporter-accelerators-collector-config\") pod \"node-exporter-pvp5f\" (UID: \"fee71c0c-b8c8-4802-b2dd-e8c2ba7389cc\") " pod="openshift-monitoring/node-exporter-pvp5f" Sep 10 13:18:21.033618 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:18:21.033595 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/fee71c0c-b8c8-4802-b2dd-e8c2ba7389cc-node-exporter-tls\") pod \"node-exporter-pvp5f\" (UID: \"fee71c0c-b8c8-4802-b2dd-e8c2ba7389cc\") " pod="openshift-monitoring/node-exporter-pvp5f" Sep 10 13:18:21.033733 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:18:21.033714 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/fee71c0c-b8c8-4802-b2dd-e8c2ba7389cc-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-pvp5f\" (UID: \"fee71c0c-b8c8-4802-b2dd-e8c2ba7389cc\") " pod="openshift-monitoring/node-exporter-pvp5f" Sep 10 13:18:21.033835 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:18:21.033815 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/9eea609d-ef83-4cdd-aacb-d0719e24b978-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-74cb57df7-dtddc\" (UID: \"9eea609d-ef83-4cdd-aacb-d0719e24b978\") " pod="openshift-monitoring/kube-state-metrics-74cb57df7-dtddc" Sep 10 13:18:21.034179 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:18:21.034151 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/f42e5bbe-1bd2-472a-b55d-786b3a65fba9-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-657fd6c54-vnznb\" (UID: \"f42e5bbe-1bd2-472a-b55d-786b3a65fba9\") " pod="openshift-monitoring/openshift-state-metrics-657fd6c54-vnznb" Sep 10 13:18:21.042910 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:18:21.042884 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rn7xz\" (UniqueName: \"kubernetes.io/projected/9eea609d-ef83-4cdd-aacb-d0719e24b978-kube-api-access-rn7xz\") pod \"kube-state-metrics-74cb57df7-dtddc\" (UID: \"9eea609d-ef83-4cdd-aacb-d0719e24b978\") " pod="openshift-monitoring/kube-state-metrics-74cb57df7-dtddc" Sep 10 13:18:21.043893 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:18:21.043871 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kmn7t\" (UniqueName: \"kubernetes.io/projected/fee71c0c-b8c8-4802-b2dd-e8c2ba7389cc-kube-api-access-kmn7t\") pod \"node-exporter-pvp5f\" (UID: \"fee71c0c-b8c8-4802-b2dd-e8c2ba7389cc\") " pod="openshift-monitoring/node-exporter-pvp5f" Sep 10 13:18:21.044677 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:18:21.044659 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8xqnd\" (UniqueName: \"kubernetes.io/projected/f42e5bbe-1bd2-472a-b55d-786b3a65fba9-kube-api-access-8xqnd\") pod \"openshift-state-metrics-657fd6c54-vnznb\" (UID: \"f42e5bbe-1bd2-472a-b55d-786b3a65fba9\") " pod="openshift-monitoring/openshift-state-metrics-657fd6c54-vnznb" Sep 10 13:18:21.183528 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:18:21.183437 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-pvp5f" Sep 10 13:18:21.453839 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:18:21.453809 2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-92n4q" Sep 10 13:18:21.536476 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:18:21.535739 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/9eea609d-ef83-4cdd-aacb-d0719e24b978-kube-state-metrics-tls\") pod \"kube-state-metrics-74cb57df7-dtddc\" (UID: \"9eea609d-ef83-4cdd-aacb-d0719e24b978\") " pod="openshift-monitoring/kube-state-metrics-74cb57df7-dtddc" Sep 10 13:18:21.536476 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:18:21.535905 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/f42e5bbe-1bd2-472a-b55d-786b3a65fba9-openshift-state-metrics-tls\") pod \"openshift-state-metrics-657fd6c54-vnznb\" (UID: \"f42e5bbe-1bd2-472a-b55d-786b3a65fba9\") " pod="openshift-monitoring/openshift-state-metrics-657fd6c54-vnznb" Sep 10 13:18:21.538748 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:18:21.538720 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/9eea609d-ef83-4cdd-aacb-d0719e24b978-kube-state-metrics-tls\") pod \"kube-state-metrics-74cb57df7-dtddc\" (UID: \"9eea609d-ef83-4cdd-aacb-d0719e24b978\") " pod="openshift-monitoring/kube-state-metrics-74cb57df7-dtddc" Sep 10 13:18:21.538885 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:18:21.538855 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/f42e5bbe-1bd2-472a-b55d-786b3a65fba9-openshift-state-metrics-tls\") pod \"openshift-state-metrics-657fd6c54-vnznb\" (UID: \"f42e5bbe-1bd2-472a-b55d-786b3a65fba9\") " pod="openshift-monitoring/openshift-state-metrics-657fd6c54-vnznb" Sep 10 13:18:21.741598 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:18:21.741505 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-657fd6c54-vnznb" Sep 10 13:18:21.750243 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:18:21.750219 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-74cb57df7-dtddc" Sep 10 13:18:21.947227 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:18:21.945213 2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Sep 10 13:18:21.951703 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:18:21.951577 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Sep 10 13:18:21.954977 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:18:21.954227 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\"" Sep 10 13:18:21.954977 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:18:21.954330 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\"" Sep 10 13:18:21.954977 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:18:21.954465 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\"" Sep 10 13:18:21.954977 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:18:21.954472 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\"" Sep 10 13:18:21.954977 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:18:21.954700 2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\"" Sep 10 13:18:21.954977 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:18:21.954758 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\"" Sep 10 13:18:21.954977 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:18:21.954875 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\"" Sep 10 13:18:21.955508 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:18:21.955354 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\"" Sep 10 13:18:21.955508 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:18:21.955385 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-r67l2\"" Sep 10 13:18:21.955508 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:18:21.955405 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\"" Sep 10 13:18:21.970390 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:18:21.970364 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Sep 10 13:18:22.041303 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:18:22.041103 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/1b94312a-a7f9-4743-8057-a5dae093befb-tls-assets\") pod \"alertmanager-main-0\" (UID: \"1b94312a-a7f9-4743-8057-a5dae093befb\") " pod="openshift-monitoring/alertmanager-main-0" Sep 10 13:18:22.041303 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:18:22.041204 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1b94312a-a7f9-4743-8057-a5dae093befb-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"1b94312a-a7f9-4743-8057-a5dae093befb\") " pod="openshift-monitoring/alertmanager-main-0" Sep 10 13:18:22.041303 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:18:22.041243 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/1b94312a-a7f9-4743-8057-a5dae093befb-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"1b94312a-a7f9-4743-8057-a5dae093befb\") " pod="openshift-monitoring/alertmanager-main-0" Sep 10 13:18:22.041303 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:18:22.041269 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/1b94312a-a7f9-4743-8057-a5dae093befb-config-volume\") pod \"alertmanager-main-0\" (UID: \"1b94312a-a7f9-4743-8057-a5dae093befb\") " pod="openshift-monitoring/alertmanager-main-0" Sep 10 13:18:22.042708 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:18:22.041330 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/1b94312a-a7f9-4743-8057-a5dae093befb-config-out\") pod \"alertmanager-main-0\" (UID: \"1b94312a-a7f9-4743-8057-a5dae093befb\") " pod="openshift-monitoring/alertmanager-main-0" Sep 10 13:18:22.042708 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:18:22.041352 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vddsr\" (UniqueName: \"kubernetes.io/projected/1b94312a-a7f9-4743-8057-a5dae093befb-kube-api-access-vddsr\") pod \"alertmanager-main-0\" (UID: \"1b94312a-a7f9-4743-8057-a5dae093befb\") " pod="openshift-monitoring/alertmanager-main-0" Sep 10 13:18:22.042708 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:18:22.041378 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/1b94312a-a7f9-4743-8057-a5dae093befb-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"1b94312a-a7f9-4743-8057-a5dae093befb\") " pod="openshift-monitoring/alertmanager-main-0" Sep 10 13:18:22.042708 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:18:22.041403 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/1b94312a-a7f9-4743-8057-a5dae093befb-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"1b94312a-a7f9-4743-8057-a5dae093befb\") " pod="openshift-monitoring/alertmanager-main-0" Sep 10 13:18:22.042708 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:18:22.041580 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/1b94312a-a7f9-4743-8057-a5dae093befb-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"1b94312a-a7f9-4743-8057-a5dae093befb\") " pod="openshift-monitoring/alertmanager-main-0" Sep 10 13:18:22.042708 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:18:22.041756 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/1b94312a-a7f9-4743-8057-a5dae093befb-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"1b94312a-a7f9-4743-8057-a5dae093befb\") " pod="openshift-monitoring/alertmanager-main-0" Sep 10 13:18:22.042708 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:18:22.041812 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/1b94312a-a7f9-4743-8057-a5dae093befb-web-config\") pod \"alertmanager-main-0\" (UID: \"1b94312a-a7f9-4743-8057-a5dae093befb\") " pod="openshift-monitoring/alertmanager-main-0" Sep 10 13:18:22.042708 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:18:22.041926 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/1b94312a-a7f9-4743-8057-a5dae093befb-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"1b94312a-a7f9-4743-8057-a5dae093befb\") " pod="openshift-monitoring/alertmanager-main-0" Sep 10 13:18:22.042708 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:18:22.041981 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/1b94312a-a7f9-4743-8057-a5dae093befb-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"1b94312a-a7f9-4743-8057-a5dae093befb\") " pod="openshift-monitoring/alertmanager-main-0" Sep 10 13:18:22.143089 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:18:22.143059 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/1b94312a-a7f9-4743-8057-a5dae093befb-tls-assets\") pod \"alertmanager-main-0\" (UID: \"1b94312a-a7f9-4743-8057-a5dae093befb\") " pod="openshift-monitoring/alertmanager-main-0" Sep 10 13:18:22.143236 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:18:22.143116 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1b94312a-a7f9-4743-8057-a5dae093befb-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"1b94312a-a7f9-4743-8057-a5dae093befb\") " pod="openshift-monitoring/alertmanager-main-0" Sep 10 13:18:22.143299 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:18:22.143246 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/1b94312a-a7f9-4743-8057-a5dae093befb-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"1b94312a-a7f9-4743-8057-a5dae093befb\") " pod="openshift-monitoring/alertmanager-main-0" Sep 10 13:18:22.143299 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:18:22.143289 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/1b94312a-a7f9-4743-8057-a5dae093befb-config-volume\") pod \"alertmanager-main-0\" (UID: \"1b94312a-a7f9-4743-8057-a5dae093befb\") " pod="openshift-monitoring/alertmanager-main-0" Sep 10 13:18:22.143385 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:18:22.143312 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/1b94312a-a7f9-4743-8057-a5dae093befb-config-out\") pod \"alertmanager-main-0\" (UID: \"1b94312a-a7f9-4743-8057-a5dae093befb\") " pod="openshift-monitoring/alertmanager-main-0" Sep 10 13:18:22.143385 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:18:22.143331 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vddsr\" (UniqueName: \"kubernetes.io/projected/1b94312a-a7f9-4743-8057-a5dae093befb-kube-api-access-vddsr\") pod \"alertmanager-main-0\" (UID: \"1b94312a-a7f9-4743-8057-a5dae093befb\") " pod="openshift-monitoring/alertmanager-main-0" Sep 10 13:18:22.143385 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:18:22.143350 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/1b94312a-a7f9-4743-8057-a5dae093befb-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"1b94312a-a7f9-4743-8057-a5dae093befb\") " pod="openshift-monitoring/alertmanager-main-0" Sep 10 13:18:22.143385 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:18:22.143366 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/1b94312a-a7f9-4743-8057-a5dae093befb-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"1b94312a-a7f9-4743-8057-a5dae093befb\") " pod="openshift-monitoring/alertmanager-main-0" Sep 10 13:18:22.143587 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:18:22.143384 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/1b94312a-a7f9-4743-8057-a5dae093befb-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"1b94312a-a7f9-4743-8057-a5dae093befb\") " pod="openshift-monitoring/alertmanager-main-0" Sep 10 13:18:22.143639 ip-10-0-139-174 kubenswrapper[2557]: E0910 13:18:22.143612 2557 secret.go:189] Couldn't get secret openshift-monitoring/alertmanager-main-tls: secret "alertmanager-main-tls" not found Sep 10 13:18:22.144663 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:18:22.143717 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/1b94312a-a7f9-4743-8057-a5dae093befb-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"1b94312a-a7f9-4743-8057-a5dae093befb\") " pod="openshift-monitoring/alertmanager-main-0" Sep 10 13:18:22.144663 ip-10-0-139-174 kubenswrapper[2557]: E0910 13:18:22.143906 2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1b94312a-a7f9-4743-8057-a5dae093befb-secret-alertmanager-main-tls podName:1b94312a-a7f9-4743-8057-a5dae093befb nodeName:}" failed. No retries permitted until 2026-09-10 13:18:22.643883728 +0000 UTC m=+62.950878122 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "secret-alertmanager-main-tls" (UniqueName: "kubernetes.io/secret/1b94312a-a7f9-4743-8057-a5dae093befb-secret-alertmanager-main-tls") pod "alertmanager-main-0" (UID: "1b94312a-a7f9-4743-8057-a5dae093befb") : secret "alertmanager-main-tls" not found Sep 10 13:18:22.144663 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:18:22.143965 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/1b94312a-a7f9-4743-8057-a5dae093befb-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"1b94312a-a7f9-4743-8057-a5dae093befb\") " pod="openshift-monitoring/alertmanager-main-0" Sep 10 13:18:22.144663 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:18:22.143759 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/1b94312a-a7f9-4743-8057-a5dae093befb-web-config\") pod \"alertmanager-main-0\" (UID: \"1b94312a-a7f9-4743-8057-a5dae093befb\") " pod="openshift-monitoring/alertmanager-main-0" Sep 10 13:18:22.144663 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:18:22.144114 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/1b94312a-a7f9-4743-8057-a5dae093befb-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"1b94312a-a7f9-4743-8057-a5dae093befb\") " pod="openshift-monitoring/alertmanager-main-0" Sep 10 13:18:22.144663 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:18:22.144162 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/1b94312a-a7f9-4743-8057-a5dae093befb-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"1b94312a-a7f9-4743-8057-a5dae093befb\") " pod="openshift-monitoring/alertmanager-main-0" Sep 10 13:18:22.144663 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:18:22.144459 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1b94312a-a7f9-4743-8057-a5dae093befb-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"1b94312a-a7f9-4743-8057-a5dae093befb\") " pod="openshift-monitoring/alertmanager-main-0" Sep 10 13:18:22.144663 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:18:22.144537 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/1b94312a-a7f9-4743-8057-a5dae093befb-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"1b94312a-a7f9-4743-8057-a5dae093befb\") " pod="openshift-monitoring/alertmanager-main-0" Sep 10 13:18:22.146626 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:18:22.146584 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/1b94312a-a7f9-4743-8057-a5dae093befb-config-volume\") pod \"alertmanager-main-0\" (UID: \"1b94312a-a7f9-4743-8057-a5dae093befb\") " pod="openshift-monitoring/alertmanager-main-0" Sep 10 13:18:22.146795 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:18:22.146763 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/1b94312a-a7f9-4743-8057-a5dae093befb-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"1b94312a-a7f9-4743-8057-a5dae093befb\") " pod="openshift-monitoring/alertmanager-main-0" Sep 10 13:18:22.147128 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:18:22.147095 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/1b94312a-a7f9-4743-8057-a5dae093befb-tls-assets\") pod \"alertmanager-main-0\" (UID: \"1b94312a-a7f9-4743-8057-a5dae093befb\") " pod="openshift-monitoring/alertmanager-main-0" Sep 10 13:18:22.147234 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:18:22.147124 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/1b94312a-a7f9-4743-8057-a5dae093befb-config-out\") pod \"alertmanager-main-0\" (UID: \"1b94312a-a7f9-4743-8057-a5dae093befb\") " pod="openshift-monitoring/alertmanager-main-0" Sep 10 13:18:22.147234 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:18:22.147171 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/1b94312a-a7f9-4743-8057-a5dae093befb-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"1b94312a-a7f9-4743-8057-a5dae093befb\") " pod="openshift-monitoring/alertmanager-main-0" Sep 10 13:18:22.147234 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:18:22.147223 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/1b94312a-a7f9-4743-8057-a5dae093befb-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"1b94312a-a7f9-4743-8057-a5dae093befb\") " pod="openshift-monitoring/alertmanager-main-0" Sep 10 13:18:22.147937 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:18:22.147919 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/1b94312a-a7f9-4743-8057-a5dae093befb-web-config\") pod \"alertmanager-main-0\" (UID: \"1b94312a-a7f9-4743-8057-a5dae093befb\") " pod="openshift-monitoring/alertmanager-main-0" Sep 10 13:18:22.148238 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:18:22.148219 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/1b94312a-a7f9-4743-8057-a5dae093befb-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"1b94312a-a7f9-4743-8057-a5dae093befb\") " pod="openshift-monitoring/alertmanager-main-0" Sep 10 13:18:22.154621 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:18:22.154602 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vddsr\" (UniqueName: \"kubernetes.io/projected/1b94312a-a7f9-4743-8057-a5dae093befb-kube-api-access-vddsr\") pod \"alertmanager-main-0\" (UID: \"1b94312a-a7f9-4743-8057-a5dae093befb\") " pod="openshift-monitoring/alertmanager-main-0" Sep 10 13:18:22.221203 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:18:22.220976 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-657fd6c54-vnznb"] Sep 10 13:18:22.224555 ip-10-0-139-174 kubenswrapper[2557]: W0910 13:18:22.224516 2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf42e5bbe_1bd2_472a_b55d_786b3a65fba9.slice/crio-76521b3516395327b945672efff5e76ffeb05513a7af20c3525ecd7f0ee30733 WatchSource:0}: Error finding container 76521b3516395327b945672efff5e76ffeb05513a7af20c3525ecd7f0ee30733: Status 404 returned error can't find the container with id 76521b3516395327b945672efff5e76ffeb05513a7af20c3525ecd7f0ee30733 Sep 10 13:18:22.225733 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:18:22.225715 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-74cb57df7-dtddc"] Sep 10 13:18:22.229082 ip-10-0-139-174 kubenswrapper[2557]: W0910 13:18:22.229060 2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9eea609d_ef83_4cdd_aacb_d0719e24b978.slice/crio-c40560a453933fa4d1c1b79087f98038b2d2999ce2c080b96aeebb716e274456 WatchSource:0}: Error finding container c40560a453933fa4d1c1b79087f98038b2d2999ce2c080b96aeebb716e274456: Status 404 returned error can't find the container with id c40560a453933fa4d1c1b79087f98038b2d2999ce2c080b96aeebb716e274456 Sep 10 13:18:22.544457 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:18:22.544351 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-pvp5f" event={"ID":"fee71c0c-b8c8-4802-b2dd-e8c2ba7389cc","Type":"ContainerStarted","Data":"33bf7741a67162f783c7c6bf52c4dfad49713b0fda59848ccc2cfd5ebf378215"} Sep 10 13:18:22.546907 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:18:22.546875 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-657fd6c54-vnznb" event={"ID":"f42e5bbe-1bd2-472a-b55d-786b3a65fba9","Type":"ContainerStarted","Data":"603595925a1c3d48f262f24f2344f2434ab02b1e714df38189a3b01e6c33cbae"} Sep 10 13:18:22.547047 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:18:22.546941 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-657fd6c54-vnznb" event={"ID":"f42e5bbe-1bd2-472a-b55d-786b3a65fba9","Type":"ContainerStarted","Data":"37eed80080f429dd41f47fbe7fe5873714d0551a8dbe498544e6551c9c539bff"} Sep 10 13:18:22.547047 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:18:22.546973 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-657fd6c54-vnznb" event={"ID":"f42e5bbe-1bd2-472a-b55d-786b3a65fba9","Type":"ContainerStarted","Data":"76521b3516395327b945672efff5e76ffeb05513a7af20c3525ecd7f0ee30733"} Sep 10 13:18:22.548271 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:18:22.548249 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-74cb57df7-dtddc" event={"ID":"9eea609d-ef83-4cdd-aacb-d0719e24b978","Type":"ContainerStarted","Data":"c40560a453933fa4d1c1b79087f98038b2d2999ce2c080b96aeebb716e274456"} Sep 10 13:18:22.550020 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:18:22.549984 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7448d46fdc-jj7z4" event={"ID":"23407ee4-a22b-4bde-81f0-899eb8532f49","Type":"ContainerStarted","Data":"6a429a033af17365bf1758c54ac20a2d8aa9f87db795c42d23d17a808ebd7ea4"} Sep 10 13:18:22.574753 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:18:22.574711 2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-7448d46fdc-jj7z4" podStartSLOduration=2.310980204 podStartE2EDuration="5.574692563s" totalImagesPullingTime="3.263712359s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 13:18:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 13:18:22.573704135 +0000 UTC m=+62.880698551" watchObservedRunningTime="2026-09-10 13:18:22.574692563 +0000 UTC m=+62.881686977" Sep 10 13:18:22.647174 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:18:22.647139 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/1b94312a-a7f9-4743-8057-a5dae093befb-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"1b94312a-a7f9-4743-8057-a5dae093befb\") " pod="openshift-monitoring/alertmanager-main-0" Sep 10 13:18:22.649933 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:18:22.649907 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/1b94312a-a7f9-4743-8057-a5dae093befb-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"1b94312a-a7f9-4743-8057-a5dae093befb\") " pod="openshift-monitoring/alertmanager-main-0" Sep 10 13:18:22.862357 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:18:22.862333 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Sep 10 13:18:23.139688 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:18:23.139561 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Sep 10 13:18:23.143737 ip-10-0-139-174 kubenswrapper[2557]: W0910 13:18:23.143697 2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1b94312a_a7f9_4743_8057_a5dae093befb.slice/crio-e9dbc24439e0b3ab25f7362e18ce06ef420558bffab84568cd626e5d64c7783f WatchSource:0}: Error finding container e9dbc24439e0b3ab25f7362e18ce06ef420558bffab84568cd626e5d64c7783f: Status 404 returned error can't find the container with id e9dbc24439e0b3ab25f7362e18ce06ef420558bffab84568cd626e5d64c7783f Sep 10 13:18:23.556019 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:18:23.555977 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-657fd6c54-vnznb" event={"ID":"f42e5bbe-1bd2-472a-b55d-786b3a65fba9","Type":"ContainerStarted","Data":"3534c09d5839b96aa29cfcc2571955ac5fa723378ff166f87305296d18a4ecf1"} Sep 10 13:18:23.557647 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:18:23.557619 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"1b94312a-a7f9-4743-8057-a5dae093befb","Type":"ContainerStarted","Data":"e9dbc24439e0b3ab25f7362e18ce06ef420558bffab84568cd626e5d64c7783f"} Sep 10 13:18:23.559589 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:18:23.559564 2557 generic.go:356] "Generic (PLEG): container finished" podID="fee71c0c-b8c8-4802-b2dd-e8c2ba7389cc" containerID="0faad8d0001cbefb86d82178890f28b46144d501e59e1b6b197cb78327d7fda4" exitCode=0 Sep 10 13:18:23.559955 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:18:23.559930 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-pvp5f" event={"ID":"fee71c0c-b8c8-4802-b2dd-e8c2ba7389cc","Type":"ContainerDied","Data":"0faad8d0001cbefb86d82178890f28b46144d501e59e1b6b197cb78327d7fda4"} Sep 10 13:18:23.603523 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:18:23.603471 2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/openshift-state-metrics-657fd6c54-vnznb" podStartSLOduration=2.465489936 podStartE2EDuration="3.603453605s" totalImagesPullingTime="1.137963669s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 13:18:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 13:18:23.602611368 +0000 UTC m=+63.909605782" watchObservedRunningTime="2026-09-10 13:18:23.603453605 +0000 UTC m=+63.910448020" Sep 10 13:18:24.565831 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:18:24.565786 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-pvp5f" event={"ID":"fee71c0c-b8c8-4802-b2dd-e8c2ba7389cc","Type":"ContainerStarted","Data":"c8e8a920de14f38b19c1a6d786f1efae4aa31170a02d3a34e773f8fd13d4a58f"} Sep 10 13:18:24.566277 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:18:24.565871 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-pvp5f" event={"ID":"fee71c0c-b8c8-4802-b2dd-e8c2ba7389cc","Type":"ContainerStarted","Data":"3b7e1b23fcbd56f0cc1b0a9e27ee2a21e26fc867abd5bb097fdefb7728fe8b4d"} Sep 10 13:18:24.569394 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:18:24.569359 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-74cb57df7-dtddc" event={"ID":"9eea609d-ef83-4cdd-aacb-d0719e24b978","Type":"ContainerStarted","Data":"60f535b3434f7b3efb75e39ddb3e174c3830fa77809d45634365ffbc0cdc1b86"} Sep 10 13:18:24.569542 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:18:24.569410 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-74cb57df7-dtddc" event={"ID":"9eea609d-ef83-4cdd-aacb-d0719e24b978","Type":"ContainerStarted","Data":"d0fed841a937a490ecf8e15ccf5ec4395e3db470eca634ba160bf932fe3db31d"} Sep 10 13:18:24.569542 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:18:24.569454 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-74cb57df7-dtddc" event={"ID":"9eea609d-ef83-4cdd-aacb-d0719e24b978","Type":"ContainerStarted","Data":"c6c8713b2da6a29e3f33abb6ca971a295c33c3e3e904c15158596b90f8dfd3bb"} Sep 10 13:18:24.635293 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:18:24.635235 2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-pvp5f" podStartSLOduration=3.726721179 podStartE2EDuration="4.635214784s" totalImagesPullingTime="835.031307ms" totalInitContainerRuntime="73.462298ms" isStatefulPod=true podCreationTimestamp="2026-09-10 13:18:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 13:18:24.60086063 +0000 UTC m=+64.907855046" watchObservedRunningTime="2026-09-10 13:18:24.635214784 +0000 UTC m=+64.942209196" Sep 10 13:18:25.274283 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:18:25.274234 2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/kube-state-metrics-74cb57df7-dtddc" podStartSLOduration=3.684900478 podStartE2EDuration="5.274216047s" totalImagesPullingTime="1.589315569s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 13:18:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 13:18:24.635074606 +0000 UTC m=+64.942069030" watchObservedRunningTime="2026-09-10 13:18:25.274216047 +0000 UTC m=+65.581210454" Sep 10 13:18:25.274744 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:18:25.274729 2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/metrics-server-6b596c7f59-v6hk8"] Sep 10 13:18:25.280649 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:18:25.280629 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-6b596c7f59-v6hk8" Sep 10 13:18:25.283117 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:18:25.283095 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-client-certs\"" Sep 10 13:18:25.283820 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:18:25.283797 2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-server-audit-profiles\"" Sep 10 13:18:25.283820 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:18:25.283816 2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\"" Sep 10 13:18:25.283999 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:18:25.283823 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-70bmnta37b1k0\"" Sep 10 13:18:25.283999 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:18:25.283806 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-tls\"" Sep 10 13:18:25.283999 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:18:25.283834 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-dockercfg-tcccd\"" Sep 10 13:18:25.289205 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:18:25.289186 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-6b596c7f59-v6hk8"] Sep 10 13:18:25.373737 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:18:25.373711 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qrfvm\" (UniqueName: \"kubernetes.io/projected/9543ae87-983f-4d7a-a2f6-36aaf9caca18-kube-api-access-qrfvm\") pod \"metrics-server-6b596c7f59-v6hk8\" (UID: \"9543ae87-983f-4d7a-a2f6-36aaf9caca18\") " pod="openshift-monitoring/metrics-server-6b596c7f59-v6hk8" Sep 10 13:18:25.373855 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:18:25.373754 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/9543ae87-983f-4d7a-a2f6-36aaf9caca18-secret-metrics-server-client-certs\") pod \"metrics-server-6b596c7f59-v6hk8\" (UID: \"9543ae87-983f-4d7a-a2f6-36aaf9caca18\") " pod="openshift-monitoring/metrics-server-6b596c7f59-v6hk8" Sep 10 13:18:25.373855 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:18:25.373831 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/9543ae87-983f-4d7a-a2f6-36aaf9caca18-secret-metrics-server-tls\") pod \"metrics-server-6b596c7f59-v6hk8\" (UID: \"9543ae87-983f-4d7a-a2f6-36aaf9caca18\") " pod="openshift-monitoring/metrics-server-6b596c7f59-v6hk8" Sep 10 13:18:25.373921 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:18:25.373859 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9543ae87-983f-4d7a-a2f6-36aaf9caca18-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-6b596c7f59-v6hk8\" (UID: \"9543ae87-983f-4d7a-a2f6-36aaf9caca18\") " pod="openshift-monitoring/metrics-server-6b596c7f59-v6hk8" Sep 10 13:18:25.373921 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:18:25.373888 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/9543ae87-983f-4d7a-a2f6-36aaf9caca18-audit-log\") pod \"metrics-server-6b596c7f59-v6hk8\" (UID: \"9543ae87-983f-4d7a-a2f6-36aaf9caca18\") " pod="openshift-monitoring/metrics-server-6b596c7f59-v6hk8" Sep 10 13:18:25.373980 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:18:25.373931 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/9543ae87-983f-4d7a-a2f6-36aaf9caca18-metrics-server-audit-profiles\") pod \"metrics-server-6b596c7f59-v6hk8\" (UID: \"9543ae87-983f-4d7a-a2f6-36aaf9caca18\") " pod="openshift-monitoring/metrics-server-6b596c7f59-v6hk8" Sep 10 13:18:25.373980 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:18:25.373954 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/9543ae87-983f-4d7a-a2f6-36aaf9caca18-client-ca-bundle\") pod \"metrics-server-6b596c7f59-v6hk8\" (UID: \"9543ae87-983f-4d7a-a2f6-36aaf9caca18\") " pod="openshift-monitoring/metrics-server-6b596c7f59-v6hk8" Sep 10 13:18:25.474381 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:18:25.474354 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/9543ae87-983f-4d7a-a2f6-36aaf9caca18-secret-metrics-server-tls\") pod \"metrics-server-6b596c7f59-v6hk8\" (UID: \"9543ae87-983f-4d7a-a2f6-36aaf9caca18\") " pod="openshift-monitoring/metrics-server-6b596c7f59-v6hk8" Sep 10 13:18:25.474381 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:18:25.474383 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9543ae87-983f-4d7a-a2f6-36aaf9caca18-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-6b596c7f59-v6hk8\" (UID: \"9543ae87-983f-4d7a-a2f6-36aaf9caca18\") " pod="openshift-monitoring/metrics-server-6b596c7f59-v6hk8" Sep 10 13:18:25.474566 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:18:25.474410 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/9543ae87-983f-4d7a-a2f6-36aaf9caca18-audit-log\") pod \"metrics-server-6b596c7f59-v6hk8\" (UID: \"9543ae87-983f-4d7a-a2f6-36aaf9caca18\") " pod="openshift-monitoring/metrics-server-6b596c7f59-v6hk8" Sep 10 13:18:25.474566 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:18:25.474447 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/9543ae87-983f-4d7a-a2f6-36aaf9caca18-metrics-server-audit-profiles\") pod \"metrics-server-6b596c7f59-v6hk8\" (UID: \"9543ae87-983f-4d7a-a2f6-36aaf9caca18\") " pod="openshift-monitoring/metrics-server-6b596c7f59-v6hk8" Sep 10 13:18:25.474566 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:18:25.474462 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/9543ae87-983f-4d7a-a2f6-36aaf9caca18-client-ca-bundle\") pod \"metrics-server-6b596c7f59-v6hk8\" (UID: \"9543ae87-983f-4d7a-a2f6-36aaf9caca18\") " pod="openshift-monitoring/metrics-server-6b596c7f59-v6hk8" Sep 10 13:18:25.474566 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:18:25.474498 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qrfvm\" (UniqueName: \"kubernetes.io/projected/9543ae87-983f-4d7a-a2f6-36aaf9caca18-kube-api-access-qrfvm\") pod \"metrics-server-6b596c7f59-v6hk8\" (UID: \"9543ae87-983f-4d7a-a2f6-36aaf9caca18\") " pod="openshift-monitoring/metrics-server-6b596c7f59-v6hk8" Sep 10 13:18:25.474566 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:18:25.474541 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/9543ae87-983f-4d7a-a2f6-36aaf9caca18-secret-metrics-server-client-certs\") pod \"metrics-server-6b596c7f59-v6hk8\" (UID: \"9543ae87-983f-4d7a-a2f6-36aaf9caca18\") " pod="openshift-monitoring/metrics-server-6b596c7f59-v6hk8" Sep 10 13:18:25.474895 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:18:25.474877 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/9543ae87-983f-4d7a-a2f6-36aaf9caca18-audit-log\") pod \"metrics-server-6b596c7f59-v6hk8\" (UID: \"9543ae87-983f-4d7a-a2f6-36aaf9caca18\") " pod="openshift-monitoring/metrics-server-6b596c7f59-v6hk8" Sep 10 13:18:25.475349 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:18:25.475326 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9543ae87-983f-4d7a-a2f6-36aaf9caca18-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-6b596c7f59-v6hk8\" (UID: \"9543ae87-983f-4d7a-a2f6-36aaf9caca18\") " pod="openshift-monitoring/metrics-server-6b596c7f59-v6hk8" Sep 10 13:18:25.475441 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:18:25.475336 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/9543ae87-983f-4d7a-a2f6-36aaf9caca18-metrics-server-audit-profiles\") pod \"metrics-server-6b596c7f59-v6hk8\" (UID: \"9543ae87-983f-4d7a-a2f6-36aaf9caca18\") " pod="openshift-monitoring/metrics-server-6b596c7f59-v6hk8" Sep 10 13:18:25.477028 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:18:25.477008 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/9543ae87-983f-4d7a-a2f6-36aaf9caca18-secret-metrics-server-tls\") pod \"metrics-server-6b596c7f59-v6hk8\" (UID: \"9543ae87-983f-4d7a-a2f6-36aaf9caca18\") " pod="openshift-monitoring/metrics-server-6b596c7f59-v6hk8" Sep 10 13:18:25.477122 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:18:25.477106 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/9543ae87-983f-4d7a-a2f6-36aaf9caca18-secret-metrics-server-client-certs\") pod \"metrics-server-6b596c7f59-v6hk8\" (UID: \"9543ae87-983f-4d7a-a2f6-36aaf9caca18\") " pod="openshift-monitoring/metrics-server-6b596c7f59-v6hk8" Sep 10 13:18:25.477180 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:18:25.477143 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/9543ae87-983f-4d7a-a2f6-36aaf9caca18-client-ca-bundle\") pod \"metrics-server-6b596c7f59-v6hk8\" (UID: \"9543ae87-983f-4d7a-a2f6-36aaf9caca18\") " pod="openshift-monitoring/metrics-server-6b596c7f59-v6hk8" Sep 10 13:18:25.482983 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:18:25.482964 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qrfvm\" (UniqueName: \"kubernetes.io/projected/9543ae87-983f-4d7a-a2f6-36aaf9caca18-kube-api-access-qrfvm\") pod \"metrics-server-6b596c7f59-v6hk8\" (UID: \"9543ae87-983f-4d7a-a2f6-36aaf9caca18\") " pod="openshift-monitoring/metrics-server-6b596c7f59-v6hk8" Sep 10 13:18:25.573805 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:18:25.573743 2557 generic.go:356] "Generic (PLEG): container finished" podID="1b94312a-a7f9-4743-8057-a5dae093befb" containerID="f30ac0076533ce6d1bfbf56b70ed2614e8c1cad5bb434e0c296a5a0a0e00a2d1" exitCode=0 Sep 10 13:18:25.573805 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:18:25.573791 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"1b94312a-a7f9-4743-8057-a5dae093befb","Type":"ContainerDied","Data":"f30ac0076533ce6d1bfbf56b70ed2614e8c1cad5bb434e0c296a5a0a0e00a2d1"} Sep 10 13:18:25.589338 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:18:25.589314 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-6b596c7f59-v6hk8" Sep 10 13:18:25.632453 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:18:25.632403 2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-7448d46fdc-jj7z4"] Sep 10 13:18:25.869525 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:18:25.869446 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-6b596c7f59-v6hk8"] Sep 10 13:18:25.872650 ip-10-0-139-174 kubenswrapper[2557]: W0910 13:18:25.872621 2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9543ae87_983f_4d7a_a2f6_36aaf9caca18.slice/crio-1c71353e68d3d37e59f9fccecad78430a6ea4d29c65e54f1af5eb3fab3f7682b WatchSource:0}: Error finding container 1c71353e68d3d37e59f9fccecad78430a6ea4d29c65e54f1af5eb3fab3f7682b: Status 404 returned error can't find the container with id 1c71353e68d3d37e59f9fccecad78430a6ea4d29c65e54f1af5eb3fab3f7682b Sep 10 13:18:25.878581 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:18:25.878559 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4016c772-d4cc-4de5-9272-f33e9da5b2b8-metrics-certs\") pod \"network-metrics-daemon-7vvkj\" (UID: \"4016c772-d4cc-4de5-9272-f33e9da5b2b8\") " pod="openshift-multus/network-metrics-daemon-7vvkj" Sep 10 13:18:25.878700 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:18:25.878625 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/d873f3a0-ccac-4d59-a6f6-2c93cf5118de-original-pull-secret\") pod \"global-pull-secret-syncer-phs5r\" (UID: \"d873f3a0-ccac-4d59-a6f6-2c93cf5118de\") " pod="kube-system/global-pull-secret-syncer-phs5r" Sep 10 13:18:25.880919 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:18:25.880901 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Sep 10 13:18:25.880985 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:18:25.880928 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Sep 10 13:18:25.891156 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:18:25.891131 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/d873f3a0-ccac-4d59-a6f6-2c93cf5118de-original-pull-secret\") pod \"global-pull-secret-syncer-phs5r\" (UID: \"d873f3a0-ccac-4d59-a6f6-2c93cf5118de\") " pod="kube-system/global-pull-secret-syncer-phs5r" Sep 10 13:18:25.891156 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:18:25.891151 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4016c772-d4cc-4de5-9272-f33e9da5b2b8-metrics-certs\") pod \"network-metrics-daemon-7vvkj\" (UID: \"4016c772-d4cc-4de5-9272-f33e9da5b2b8\") " pod="openshift-multus/network-metrics-daemon-7vvkj" Sep 10 13:18:25.979468 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:18:25.979443 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r6pw5\" (UniqueName: \"kubernetes.io/projected/accbb486-a5c9-469d-beca-837d2bbd4eab-kube-api-access-r6pw5\") pod \"network-check-target-6ggtx\" (UID: \"accbb486-a5c9-469d-beca-837d2bbd4eab\") " pod="openshift-network-diagnostics/network-check-target-6ggtx" Sep 10 13:18:25.981736 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:18:25.981721 2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Sep 10 13:18:25.992854 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:18:25.992840 2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Sep 10 13:18:26.003485 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:18:26.003464 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-r6pw5\" (UniqueName: \"kubernetes.io/projected/accbb486-a5c9-469d-beca-837d2bbd4eab-kube-api-access-r6pw5\") pod \"network-check-target-6ggtx\" (UID: \"accbb486-a5c9-469d-beca-837d2bbd4eab\") " pod="openshift-network-diagnostics/network-check-target-6ggtx" Sep 10 13:18:26.037725 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:18:26.037707 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-w47rx\"" Sep 10 13:18:26.038996 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:18:26.038979 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-5vlq4\"" Sep 10 13:18:26.039498 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:18:26.039485 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-6ggtx" Sep 10 13:18:26.041752 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:18:26.041738 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-phs5r" Sep 10 13:18:26.046860 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:18:26.046846 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-7vvkj" Sep 10 13:18:26.112312 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:18:26.111902 2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/telemeter-client-6ccc84c476-zwhn8"] Sep 10 13:18:26.140454 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:18:26.139981 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-6ccc84c476-zwhn8"] Sep 10 13:18:26.140454 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:18:26.140160 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-6ccc84c476-zwhn8" Sep 10 13:18:26.143468 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:18:26.143173 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client\"" Sep 10 13:18:26.147324 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:18:26.144160 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-kube-rbac-proxy-config\"" Sep 10 13:18:26.151230 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:18:26.148015 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"federate-client-certs\"" Sep 10 13:18:26.151230 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:18:26.148822 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-dockercfg-xckf8\"" Sep 10 13:18:26.151230 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:18:26.149028 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-tls\"" Sep 10 13:18:26.151230 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:18:26.149779 2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-client-serving-certs-ca-bundle\"" Sep 10 13:18:26.173729 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:18:26.169334 2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-trusted-ca-bundle-8i12ta5c71j38\"" Sep 10 13:18:26.181366 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:18:26.181270 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/222b4b70-7a97-4b6e-9f37-840e45dab882-telemeter-client-tls\") pod \"telemeter-client-6ccc84c476-zwhn8\" (UID: \"222b4b70-7a97-4b6e-9f37-840e45dab882\") " pod="openshift-monitoring/telemeter-client-6ccc84c476-zwhn8" Sep 10 13:18:26.181366 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:18:26.181310 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/222b4b70-7a97-4b6e-9f37-840e45dab882-metrics-client-ca\") pod \"telemeter-client-6ccc84c476-zwhn8\" (UID: \"222b4b70-7a97-4b6e-9f37-840e45dab882\") " pod="openshift-monitoring/telemeter-client-6ccc84c476-zwhn8" Sep 10 13:18:26.181366 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:18:26.181343 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/222b4b70-7a97-4b6e-9f37-840e45dab882-secret-telemeter-client\") pod \"telemeter-client-6ccc84c476-zwhn8\" (UID: \"222b4b70-7a97-4b6e-9f37-840e45dab882\") " pod="openshift-monitoring/telemeter-client-6ccc84c476-zwhn8" Sep 10 13:18:26.181366 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:18:26.181369 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jzd7j\" (UniqueName: \"kubernetes.io/projected/222b4b70-7a97-4b6e-9f37-840e45dab882-kube-api-access-jzd7j\") pod \"telemeter-client-6ccc84c476-zwhn8\" (UID: \"222b4b70-7a97-4b6e-9f37-840e45dab882\") " pod="openshift-monitoring/telemeter-client-6ccc84c476-zwhn8" Sep 10 13:18:26.181641 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:18:26.181436 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/222b4b70-7a97-4b6e-9f37-840e45dab882-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-6ccc84c476-zwhn8\" (UID: \"222b4b70-7a97-4b6e-9f37-840e45dab882\") " pod="openshift-monitoring/telemeter-client-6ccc84c476-zwhn8" Sep 10 13:18:26.181641 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:18:26.181466 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/222b4b70-7a97-4b6e-9f37-840e45dab882-telemeter-trusted-ca-bundle\") pod \"telemeter-client-6ccc84c476-zwhn8\" (UID: \"222b4b70-7a97-4b6e-9f37-840e45dab882\") " pod="openshift-monitoring/telemeter-client-6ccc84c476-zwhn8" Sep 10 13:18:26.181641 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:18:26.181544 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/222b4b70-7a97-4b6e-9f37-840e45dab882-federate-client-tls\") pod \"telemeter-client-6ccc84c476-zwhn8\" (UID: \"222b4b70-7a97-4b6e-9f37-840e45dab882\") " pod="openshift-monitoring/telemeter-client-6ccc84c476-zwhn8" Sep 10 13:18:26.181641 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:18:26.181583 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/222b4b70-7a97-4b6e-9f37-840e45dab882-serving-certs-ca-bundle\") pod \"telemeter-client-6ccc84c476-zwhn8\" (UID: \"222b4b70-7a97-4b6e-9f37-840e45dab882\") " pod="openshift-monitoring/telemeter-client-6ccc84c476-zwhn8" Sep 10 13:18:26.282512 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:18:26.282476 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/222b4b70-7a97-4b6e-9f37-840e45dab882-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-6ccc84c476-zwhn8\" (UID: \"222b4b70-7a97-4b6e-9f37-840e45dab882\") " pod="openshift-monitoring/telemeter-client-6ccc84c476-zwhn8" Sep 10 13:18:26.282698 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:18:26.282521 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/222b4b70-7a97-4b6e-9f37-840e45dab882-telemeter-trusted-ca-bundle\") pod \"telemeter-client-6ccc84c476-zwhn8\" (UID: \"222b4b70-7a97-4b6e-9f37-840e45dab882\") " pod="openshift-monitoring/telemeter-client-6ccc84c476-zwhn8" Sep 10 13:18:26.282698 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:18:26.282606 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/222b4b70-7a97-4b6e-9f37-840e45dab882-federate-client-tls\") pod \"telemeter-client-6ccc84c476-zwhn8\" (UID: \"222b4b70-7a97-4b6e-9f37-840e45dab882\") " pod="openshift-monitoring/telemeter-client-6ccc84c476-zwhn8" Sep 10 13:18:26.282698 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:18:26.282643 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/222b4b70-7a97-4b6e-9f37-840e45dab882-serving-certs-ca-bundle\") pod \"telemeter-client-6ccc84c476-zwhn8\" (UID: \"222b4b70-7a97-4b6e-9f37-840e45dab882\") " pod="openshift-monitoring/telemeter-client-6ccc84c476-zwhn8" Sep 10 13:18:26.282698 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:18:26.282686 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/222b4b70-7a97-4b6e-9f37-840e45dab882-telemeter-client-tls\") pod \"telemeter-client-6ccc84c476-zwhn8\" (UID: \"222b4b70-7a97-4b6e-9f37-840e45dab882\") " pod="openshift-monitoring/telemeter-client-6ccc84c476-zwhn8" Sep 10 13:18:26.282930 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:18:26.282709 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/222b4b70-7a97-4b6e-9f37-840e45dab882-metrics-client-ca\") pod \"telemeter-client-6ccc84c476-zwhn8\" (UID: \"222b4b70-7a97-4b6e-9f37-840e45dab882\") " pod="openshift-monitoring/telemeter-client-6ccc84c476-zwhn8" Sep 10 13:18:26.283232 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:18:26.283190 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/222b4b70-7a97-4b6e-9f37-840e45dab882-secret-telemeter-client\") pod \"telemeter-client-6ccc84c476-zwhn8\" (UID: \"222b4b70-7a97-4b6e-9f37-840e45dab882\") " pod="openshift-monitoring/telemeter-client-6ccc84c476-zwhn8" Sep 10 13:18:26.283372 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:18:26.283273 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jzd7j\" (UniqueName: \"kubernetes.io/projected/222b4b70-7a97-4b6e-9f37-840e45dab882-kube-api-access-jzd7j\") pod \"telemeter-client-6ccc84c476-zwhn8\" (UID: \"222b4b70-7a97-4b6e-9f37-840e45dab882\") " pod="openshift-monitoring/telemeter-client-6ccc84c476-zwhn8" Sep 10 13:18:26.284131 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:18:26.284092 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/222b4b70-7a97-4b6e-9f37-840e45dab882-serving-certs-ca-bundle\") pod \"telemeter-client-6ccc84c476-zwhn8\" (UID: \"222b4b70-7a97-4b6e-9f37-840e45dab882\") " pod="openshift-monitoring/telemeter-client-6ccc84c476-zwhn8" Sep 10 13:18:26.284221 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:18:26.284182 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/222b4b70-7a97-4b6e-9f37-840e45dab882-telemeter-trusted-ca-bundle\") pod \"telemeter-client-6ccc84c476-zwhn8\" (UID: \"222b4b70-7a97-4b6e-9f37-840e45dab882\") " pod="openshift-monitoring/telemeter-client-6ccc84c476-zwhn8" Sep 10 13:18:26.286675 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:18:26.284617 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/222b4b70-7a97-4b6e-9f37-840e45dab882-metrics-client-ca\") pod \"telemeter-client-6ccc84c476-zwhn8\" (UID: \"222b4b70-7a97-4b6e-9f37-840e45dab882\") " pod="openshift-monitoring/telemeter-client-6ccc84c476-zwhn8" Sep 10 13:18:26.286675 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:18:26.286490 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/222b4b70-7a97-4b6e-9f37-840e45dab882-federate-client-tls\") pod \"telemeter-client-6ccc84c476-zwhn8\" (UID: \"222b4b70-7a97-4b6e-9f37-840e45dab882\") " pod="openshift-monitoring/telemeter-client-6ccc84c476-zwhn8" Sep 10 13:18:26.288479 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:18:26.286822 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/222b4b70-7a97-4b6e-9f37-840e45dab882-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-6ccc84c476-zwhn8\" (UID: \"222b4b70-7a97-4b6e-9f37-840e45dab882\") " pod="openshift-monitoring/telemeter-client-6ccc84c476-zwhn8" Sep 10 13:18:26.288479 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:18:26.287120 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/222b4b70-7a97-4b6e-9f37-840e45dab882-secret-telemeter-client\") pod \"telemeter-client-6ccc84c476-zwhn8\" (UID: \"222b4b70-7a97-4b6e-9f37-840e45dab882\") " pod="openshift-monitoring/telemeter-client-6ccc84c476-zwhn8" Sep 10 13:18:26.290260 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:18:26.290238 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/222b4b70-7a97-4b6e-9f37-840e45dab882-telemeter-client-tls\") pod \"telemeter-client-6ccc84c476-zwhn8\" (UID: \"222b4b70-7a97-4b6e-9f37-840e45dab882\") " pod="openshift-monitoring/telemeter-client-6ccc84c476-zwhn8" Sep 10 13:18:26.293147 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:18:26.293128 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jzd7j\" (UniqueName: \"kubernetes.io/projected/222b4b70-7a97-4b6e-9f37-840e45dab882-kube-api-access-jzd7j\") pod \"telemeter-client-6ccc84c476-zwhn8\" (UID: \"222b4b70-7a97-4b6e-9f37-840e45dab882\") " pod="openshift-monitoring/telemeter-client-6ccc84c476-zwhn8" Sep 10 13:18:26.337388 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:18:26.337298 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-6ggtx"] Sep 10 13:18:26.339796 ip-10-0-139-174 kubenswrapper[2557]: W0910 13:18:26.339768 2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaccbb486_a5c9_469d_beca_837d2bbd4eab.slice/crio-61ae2b840453a026f5aaadb6516223211a3775dc79e330f947e2c854081ca9c2 WatchSource:0}: Error finding container 61ae2b840453a026f5aaadb6516223211a3775dc79e330f947e2c854081ca9c2: Status 404 returned error can't find the container with id 61ae2b840453a026f5aaadb6516223211a3775dc79e330f947e2c854081ca9c2 Sep 10 13:18:26.380825 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:18:26.380772 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-phs5r"] Sep 10 13:18:26.383158 ip-10-0-139-174 kubenswrapper[2557]: W0910 13:18:26.383139 2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd873f3a0_ccac_4d59_a6f6_2c93cf5118de.slice/crio-5f9e3f8d5d636fe0db332154ec6412321697ab5972bc0ebc4b4a3638e02ea8e4 WatchSource:0}: Error finding container 5f9e3f8d5d636fe0db332154ec6412321697ab5972bc0ebc4b4a3638e02ea8e4: Status 404 returned error can't find the container with id 5f9e3f8d5d636fe0db332154ec6412321697ab5972bc0ebc4b4a3638e02ea8e4 Sep 10 13:18:26.412835 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:18:26.412811 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-7vvkj"] Sep 10 13:18:26.415351 ip-10-0-139-174 kubenswrapper[2557]: W0910 13:18:26.415331 2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4016c772_d4cc_4de5_9272_f33e9da5b2b8.slice/crio-b663619a57a25d3f6377cf561d4d1247858566dd06490961333ff3a15cb7497d WatchSource:0}: Error finding container b663619a57a25d3f6377cf561d4d1247858566dd06490961333ff3a15cb7497d: Status 404 returned error can't find the container with id b663619a57a25d3f6377cf561d4d1247858566dd06490961333ff3a15cb7497d Sep 10 13:18:26.475636 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:18:26.475614 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-6ccc84c476-zwhn8" Sep 10 13:18:26.579591 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:18:26.579560 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-7vvkj" event={"ID":"4016c772-d4cc-4de5-9272-f33e9da5b2b8","Type":"ContainerStarted","Data":"b663619a57a25d3f6377cf561d4d1247858566dd06490961333ff3a15cb7497d"} Sep 10 13:18:26.580862 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:18:26.580839 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-6b596c7f59-v6hk8" event={"ID":"9543ae87-983f-4d7a-a2f6-36aaf9caca18","Type":"ContainerStarted","Data":"1c71353e68d3d37e59f9fccecad78430a6ea4d29c65e54f1af5eb3fab3f7682b"} Sep 10 13:18:26.581949 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:18:26.581929 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-phs5r" event={"ID":"d873f3a0-ccac-4d59-a6f6-2c93cf5118de","Type":"ContainerStarted","Data":"5f9e3f8d5d636fe0db332154ec6412321697ab5972bc0ebc4b4a3638e02ea8e4"} Sep 10 13:18:26.583013 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:18:26.582993 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-6ggtx" event={"ID":"accbb486-a5c9-469d-beca-837d2bbd4eab","Type":"ContainerStarted","Data":"61ae2b840453a026f5aaadb6516223211a3775dc79e330f947e2c854081ca9c2"} Sep 10 13:18:26.782827 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:18:26.782803 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-6ccc84c476-zwhn8"] Sep 10 13:18:26.786083 ip-10-0-139-174 kubenswrapper[2557]: W0910 13:18:26.786053 2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod222b4b70_7a97_4b6e_9f37_840e45dab882.slice/crio-c1a32c4953dcbf7d2426de41cb69f6d14388394cae60dca527be9e2253873708 WatchSource:0}: Error finding container c1a32c4953dcbf7d2426de41cb69f6d14388394cae60dca527be9e2253873708: Status 404 returned error can't find the container with id c1a32c4953dcbf7d2426de41cb69f6d14388394cae60dca527be9e2253873708 Sep 10 13:18:27.590293 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:18:27.590255 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-6ccc84c476-zwhn8" event={"ID":"222b4b70-7a97-4b6e-9f37-840e45dab882","Type":"ContainerStarted","Data":"c1a32c4953dcbf7d2426de41cb69f6d14388394cae60dca527be9e2253873708"} Sep 10 13:18:28.128384 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:18:28.128347 2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-7448d46fdc-jj7z4" Sep 10 13:18:29.607510 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:18:29.607478 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"1b94312a-a7f9-4743-8057-a5dae093befb","Type":"ContainerStarted","Data":"8a1ba4a366a39a1d0834a8c2111525246b66800b9f8e54e432d75b2c72b4ae66"} Sep 10 13:18:29.607938 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:18:29.607538 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"1b94312a-a7f9-4743-8057-a5dae093befb","Type":"ContainerStarted","Data":"4d5c948938735bf7898acbabf4a14a44db170c609d0930457d158212483741f6"} Sep 10 13:18:29.607938 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:18:29.607564 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"1b94312a-a7f9-4743-8057-a5dae093befb","Type":"ContainerStarted","Data":"08d07cb07e5418ef818ba260e3db681fd3a3d4427e5d3c4e9de6e447d4c7c2fd"} Sep 10 13:18:29.607938 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:18:29.607588 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"1b94312a-a7f9-4743-8057-a5dae093befb","Type":"ContainerStarted","Data":"5bd13798b08874bab211fe5a06d81fda9792013d4277386570bdf8a81e162ae1"} Sep 10 13:18:29.607938 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:18:29.607610 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"1b94312a-a7f9-4743-8057-a5dae093befb","Type":"ContainerStarted","Data":"9f6ee9b7d6a4567cc2311feb9a9101aabad8634cbcca8e9a8b02025c3a9791af"} Sep 10 13:18:29.610836 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:18:29.610806 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-7vvkj" event={"ID":"4016c772-d4cc-4de5-9272-f33e9da5b2b8","Type":"ContainerStarted","Data":"bc8790bca64001373d251dc046abaff0f9fa9136b11f464abff3856103c8a398"} Sep 10 13:18:29.610924 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:18:29.610866 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-7vvkj" event={"ID":"4016c772-d4cc-4de5-9272-f33e9da5b2b8","Type":"ContainerStarted","Data":"67a1aa5d84a1f6804ae52896c15c537cba369b9b38d8b7e5a01816229930b269"} Sep 10 13:18:29.613923 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:18:29.613873 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-6b596c7f59-v6hk8" event={"ID":"9543ae87-983f-4d7a-a2f6-36aaf9caca18","Type":"ContainerStarted","Data":"9a413f7af2ff0ca83aa18398fdbf0c8dc7423c36b42ccb92ce81d95dbf27ddc3"} Sep 10 13:18:29.639897 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:18:29.639833 2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-7vvkj" podStartSLOduration=67.493964956 podStartE2EDuration="1m9.63981382s" totalImagesPullingTime="2.145848864s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 13:17:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 13:18:29.638981236 +0000 UTC m=+69.945975653" watchObservedRunningTime="2026-09-10 13:18:29.63981382 +0000 UTC m=+69.946808234" Sep 10 13:18:29.663178 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:18:29.663118 2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/metrics-server-6b596c7f59-v6hk8" podStartSLOduration=2.007537966 podStartE2EDuration="4.663098999s" totalImagesPullingTime="2.655561033s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 13:18:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 13:18:29.661587387 +0000 UTC m=+69.968581801" watchObservedRunningTime="2026-09-10 13:18:29.663098999 +0000 UTC m=+69.970093426" Sep 10 13:18:30.541210 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:18:30.541182 2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-9x5xs" Sep 10 13:18:32.630194 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:18:32.630108 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"1b94312a-a7f9-4743-8057-a5dae093befb","Type":"ContainerStarted","Data":"1cf9d39367a41907d3e7988b6a28b15562617005e2909d44160c84d7e5f0733a"} Sep 10 13:18:32.631745 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:18:32.631713 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-phs5r" event={"ID":"d873f3a0-ccac-4d59-a6f6-2c93cf5118de","Type":"ContainerStarted","Data":"468e223e8b865289a937d5fa2f560a89a8a513263677458082f623c97e0be897"} Sep 10 13:18:32.633240 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:18:32.633216 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-6ggtx" event={"ID":"accbb486-a5c9-469d-beca-837d2bbd4eab","Type":"ContainerStarted","Data":"44159c73b557dfc377db7fca9503bdac389012addf55646996ff1cf2f89b290c"} Sep 10 13:18:32.633361 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:18:32.633276 2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-6ggtx" Sep 10 13:18:32.635269 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:18:32.635246 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-6ccc84c476-zwhn8" event={"ID":"222b4b70-7a97-4b6e-9f37-840e45dab882","Type":"ContainerStarted","Data":"cdc23077c78cdf191ad4943d03b35d46d12d6a31995de307832ac0b9193c7f9f"} Sep 10 13:18:32.635339 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:18:32.635295 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-6ccc84c476-zwhn8" event={"ID":"222b4b70-7a97-4b6e-9f37-840e45dab882","Type":"ContainerStarted","Data":"e12381f9a2742112dea2c1e2900898e55cb7753d35d849524b9e658058d61a49"} Sep 10 13:18:32.635371 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:18:32.635339 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-6ccc84c476-zwhn8" event={"ID":"222b4b70-7a97-4b6e-9f37-840e45dab882","Type":"ContainerStarted","Data":"72db7044f19851338a0eb0ad9560af6ab17b8195297e9a06ba3e9317d577c763"} Sep 10 13:18:32.670522 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:18:32.670483 2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/alertmanager-main-0" podStartSLOduration=3.953275758 podStartE2EDuration="11.670469078s" totalImagesPullingTime="7.698829139s" totalInitContainerRuntime="18.364181ms" isStatefulPod=false podCreationTimestamp="2026-09-10 13:18:21 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 13:18:32.6704511 +0000 UTC m=+72.977445515" watchObservedRunningTime="2026-09-10 13:18:32.670469078 +0000 UTC m=+72.977463483" Sep 10 13:18:32.704972 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:18:32.704899 2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-6ggtx" podStartSLOduration=66.907288243 podStartE2EDuration="1m12.704881599s" totalImagesPullingTime="5.797593356s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 13:17:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 13:18:32.704371447 +0000 UTC m=+73.011365862" watchObservedRunningTime="2026-09-10 13:18:32.704881599 +0000 UTC m=+73.011876015" Sep 10 13:18:32.705938 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:18:32.705897 2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-phs5r" podStartSLOduration=66.949881612 podStartE2EDuration="1m12.705885267s" totalImagesPullingTime="5.756003655s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-10 13:17:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 13:18:32.688453998 +0000 UTC m=+72.995448411" watchObservedRunningTime="2026-09-10 13:18:32.705885267 +0000 UTC m=+73.012879682" Sep 10 13:18:32.729273 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:18:32.729213 2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/telemeter-client-6ccc84c476-zwhn8" podStartSLOduration=1.376632076 podStartE2EDuration="6.729196293s" totalImagesPullingTime="5.352564217s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 13:18:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 13:18:32.728504974 +0000 UTC m=+73.035499399" watchObservedRunningTime="2026-09-10 13:18:32.729196293 +0000 UTC m=+73.036190708" Sep 10 13:18:32.937764 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:18:32.937698 2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-6c9ff75b79-gzl7j"] Sep 10 13:18:32.940963 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:18:32.940940 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6c9ff75b79-gzl7j" Sep 10 13:18:32.975462 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:18:32.975434 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-6c9ff75b79-gzl7j"] Sep 10 13:18:33.039370 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:18:33.039343 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7tvk4\" (UniqueName: \"kubernetes.io/projected/ad02681a-5dfa-4abd-9ee4-6937c1c6e26a-kube-api-access-7tvk4\") pod \"console-6c9ff75b79-gzl7j\" (UID: \"ad02681a-5dfa-4abd-9ee4-6937c1c6e26a\") " pod="openshift-console/console-6c9ff75b79-gzl7j" Sep 10 13:18:33.039514 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:18:33.039381 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ad02681a-5dfa-4abd-9ee4-6937c1c6e26a-trusted-ca-bundle\") pod \"console-6c9ff75b79-gzl7j\" (UID: \"ad02681a-5dfa-4abd-9ee4-6937c1c6e26a\") " pod="openshift-console/console-6c9ff75b79-gzl7j" Sep 10 13:18:33.039514 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:18:33.039404 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/ad02681a-5dfa-4abd-9ee4-6937c1c6e26a-console-serving-cert\") pod \"console-6c9ff75b79-gzl7j\" (UID: \"ad02681a-5dfa-4abd-9ee4-6937c1c6e26a\") " pod="openshift-console/console-6c9ff75b79-gzl7j" Sep 10 13:18:33.039514 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:18:33.039442 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/ad02681a-5dfa-4abd-9ee4-6937c1c6e26a-oauth-serving-cert\") pod \"console-6c9ff75b79-gzl7j\" (UID: \"ad02681a-5dfa-4abd-9ee4-6937c1c6e26a\") " pod="openshift-console/console-6c9ff75b79-gzl7j" Sep 10 13:18:33.039514 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:18:33.039501 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/ad02681a-5dfa-4abd-9ee4-6937c1c6e26a-console-oauth-config\") pod \"console-6c9ff75b79-gzl7j\" (UID: \"ad02681a-5dfa-4abd-9ee4-6937c1c6e26a\") " pod="openshift-console/console-6c9ff75b79-gzl7j" Sep 10 13:18:33.039652 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:18:33.039549 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/ad02681a-5dfa-4abd-9ee4-6937c1c6e26a-service-ca\") pod \"console-6c9ff75b79-gzl7j\" (UID: \"ad02681a-5dfa-4abd-9ee4-6937c1c6e26a\") " pod="openshift-console/console-6c9ff75b79-gzl7j" Sep 10 13:18:33.039652 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:18:33.039578 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/ad02681a-5dfa-4abd-9ee4-6937c1c6e26a-console-config\") pod \"console-6c9ff75b79-gzl7j\" (UID: \"ad02681a-5dfa-4abd-9ee4-6937c1c6e26a\") " pod="openshift-console/console-6c9ff75b79-gzl7j" Sep 10 13:18:33.140812 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:18:33.140780 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/ad02681a-5dfa-4abd-9ee4-6937c1c6e26a-console-oauth-config\") pod \"console-6c9ff75b79-gzl7j\" (UID: \"ad02681a-5dfa-4abd-9ee4-6937c1c6e26a\") " pod="openshift-console/console-6c9ff75b79-gzl7j" Sep 10 13:18:33.140938 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:18:33.140822 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/ad02681a-5dfa-4abd-9ee4-6937c1c6e26a-service-ca\") pod \"console-6c9ff75b79-gzl7j\" (UID: \"ad02681a-5dfa-4abd-9ee4-6937c1c6e26a\") " pod="openshift-console/console-6c9ff75b79-gzl7j" Sep 10 13:18:33.140938 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:18:33.140844 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/ad02681a-5dfa-4abd-9ee4-6937c1c6e26a-console-config\") pod \"console-6c9ff75b79-gzl7j\" (UID: \"ad02681a-5dfa-4abd-9ee4-6937c1c6e26a\") " pod="openshift-console/console-6c9ff75b79-gzl7j" Sep 10 13:18:33.140938 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:18:33.140862 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7tvk4\" (UniqueName: \"kubernetes.io/projected/ad02681a-5dfa-4abd-9ee4-6937c1c6e26a-kube-api-access-7tvk4\") pod \"console-6c9ff75b79-gzl7j\" (UID: \"ad02681a-5dfa-4abd-9ee4-6937c1c6e26a\") " pod="openshift-console/console-6c9ff75b79-gzl7j" Sep 10 13:18:33.141069 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:18:33.140977 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ad02681a-5dfa-4abd-9ee4-6937c1c6e26a-trusted-ca-bundle\") pod \"console-6c9ff75b79-gzl7j\" (UID: \"ad02681a-5dfa-4abd-9ee4-6937c1c6e26a\") " pod="openshift-console/console-6c9ff75b79-gzl7j" Sep 10 13:18:33.141069 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:18:33.141021 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/ad02681a-5dfa-4abd-9ee4-6937c1c6e26a-console-serving-cert\") pod \"console-6c9ff75b79-gzl7j\" (UID: \"ad02681a-5dfa-4abd-9ee4-6937c1c6e26a\") " pod="openshift-console/console-6c9ff75b79-gzl7j" Sep 10 13:18:33.141069 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:18:33.141049 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/ad02681a-5dfa-4abd-9ee4-6937c1c6e26a-oauth-serving-cert\") pod \"console-6c9ff75b79-gzl7j\" (UID: \"ad02681a-5dfa-4abd-9ee4-6937c1c6e26a\") " pod="openshift-console/console-6c9ff75b79-gzl7j" Sep 10 13:18:33.141705 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:18:33.141679 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/ad02681a-5dfa-4abd-9ee4-6937c1c6e26a-service-ca\") pod \"console-6c9ff75b79-gzl7j\" (UID: \"ad02681a-5dfa-4abd-9ee4-6937c1c6e26a\") " pod="openshift-console/console-6c9ff75b79-gzl7j" Sep 10 13:18:33.141831 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:18:33.141811 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ad02681a-5dfa-4abd-9ee4-6937c1c6e26a-trusted-ca-bundle\") pod \"console-6c9ff75b79-gzl7j\" (UID: \"ad02681a-5dfa-4abd-9ee4-6937c1c6e26a\") " pod="openshift-console/console-6c9ff75b79-gzl7j" Sep 10 13:18:33.141909 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:18:33.141856 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/ad02681a-5dfa-4abd-9ee4-6937c1c6e26a-console-config\") pod \"console-6c9ff75b79-gzl7j\" (UID: \"ad02681a-5dfa-4abd-9ee4-6937c1c6e26a\") " pod="openshift-console/console-6c9ff75b79-gzl7j" Sep 10 13:18:33.141909 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:18:33.141861 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/ad02681a-5dfa-4abd-9ee4-6937c1c6e26a-oauth-serving-cert\") pod \"console-6c9ff75b79-gzl7j\" (UID: \"ad02681a-5dfa-4abd-9ee4-6937c1c6e26a\") " pod="openshift-console/console-6c9ff75b79-gzl7j" Sep 10 13:18:33.143285 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:18:33.143268 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/ad02681a-5dfa-4abd-9ee4-6937c1c6e26a-console-oauth-config\") pod \"console-6c9ff75b79-gzl7j\" (UID: \"ad02681a-5dfa-4abd-9ee4-6937c1c6e26a\") " pod="openshift-console/console-6c9ff75b79-gzl7j" Sep 10 13:18:33.143431 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:18:33.143401 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/ad02681a-5dfa-4abd-9ee4-6937c1c6e26a-console-serving-cert\") pod \"console-6c9ff75b79-gzl7j\" (UID: \"ad02681a-5dfa-4abd-9ee4-6937c1c6e26a\") " pod="openshift-console/console-6c9ff75b79-gzl7j" Sep 10 13:18:33.152501 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:18:33.152484 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7tvk4\" (UniqueName: \"kubernetes.io/projected/ad02681a-5dfa-4abd-9ee4-6937c1c6e26a-kube-api-access-7tvk4\") pod \"console-6c9ff75b79-gzl7j\" (UID: \"ad02681a-5dfa-4abd-9ee4-6937c1c6e26a\") " pod="openshift-console/console-6c9ff75b79-gzl7j" Sep 10 13:18:33.250001 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:18:33.249975 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6c9ff75b79-gzl7j" Sep 10 13:18:33.534545 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:18:33.534470 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-6c9ff75b79-gzl7j"] Sep 10 13:18:33.537799 ip-10-0-139-174 kubenswrapper[2557]: W0910 13:18:33.537769 2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podad02681a_5dfa_4abd_9ee4_6937c1c6e26a.slice/crio-84b8ee716bd8883708be9b3e51822a449c8e7cc7026761bf6733cb29e6990dcd WatchSource:0}: Error finding container 84b8ee716bd8883708be9b3e51822a449c8e7cc7026761bf6733cb29e6990dcd: Status 404 returned error can't find the container with id 84b8ee716bd8883708be9b3e51822a449c8e7cc7026761bf6733cb29e6990dcd Sep 10 13:18:33.641456 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:18:33.641402 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6c9ff75b79-gzl7j" event={"ID":"ad02681a-5dfa-4abd-9ee4-6937c1c6e26a","Type":"ContainerStarted","Data":"8f3e3372567439a923d097ab61190d67272e1791829ee7256bec21da74eabaed"} Sep 10 13:18:33.641890 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:18:33.641478 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6c9ff75b79-gzl7j" event={"ID":"ad02681a-5dfa-4abd-9ee4-6937c1c6e26a","Type":"ContainerStarted","Data":"84b8ee716bd8883708be9b3e51822a449c8e7cc7026761bf6733cb29e6990dcd"} Sep 10 13:18:33.663053 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:18:33.663000 2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-6c9ff75b79-gzl7j" podStartSLOduration=1.662983621 podStartE2EDuration="1.662983621s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 13:18:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 13:18:33.662278942 +0000 UTC m=+73.969273355" watchObservedRunningTime="2026-09-10 13:18:33.662983621 +0000 UTC m=+73.969978033" Sep 10 13:18:43.251178 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:18:43.251037 2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-6c9ff75b79-gzl7j" Sep 10 13:18:43.251178 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:18:43.251122 2557 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-6c9ff75b79-gzl7j" Sep 10 13:18:43.255646 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:18:43.255626 2557 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-6c9ff75b79-gzl7j" Sep 10 13:18:43.682458 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:18:43.682350 2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-6c9ff75b79-gzl7j" Sep 10 13:18:45.589838 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:18:45.589805 2557 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/metrics-server-6b596c7f59-v6hk8" Sep 10 13:18:45.590228 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:18:45.589871 2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/metrics-server-6b596c7f59-v6hk8" Sep 10 13:18:50.660572 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:18:50.660527 2557 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-7448d46fdc-jj7z4" podUID="23407ee4-a22b-4bde-81f0-899eb8532f49" containerName="console" containerID="cri-o://6a429a033af17365bf1758c54ac20a2d8aa9f87db795c42d23d17a808ebd7ea4" gracePeriod=15 Sep 10 13:18:50.904021 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:18:50.903995 2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7448d46fdc-jj7z4" Sep 10 13:18:50.963488 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:18:50.963463 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/23407ee4-a22b-4bde-81f0-899eb8532f49-oauth-serving-cert\") pod \"23407ee4-a22b-4bde-81f0-899eb8532f49\" (UID: \"23407ee4-a22b-4bde-81f0-899eb8532f49\") " Sep 10 13:18:50.963616 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:18:50.963502 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/23407ee4-a22b-4bde-81f0-899eb8532f49-console-config\") pod \"23407ee4-a22b-4bde-81f0-899eb8532f49\" (UID: \"23407ee4-a22b-4bde-81f0-899eb8532f49\") " Sep 10 13:18:50.963616 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:18:50.963529 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-75ww7\" (UniqueName: \"kubernetes.io/projected/23407ee4-a22b-4bde-81f0-899eb8532f49-kube-api-access-75ww7\") pod \"23407ee4-a22b-4bde-81f0-899eb8532f49\" (UID: \"23407ee4-a22b-4bde-81f0-899eb8532f49\") " Sep 10 13:18:50.963616 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:18:50.963565 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/23407ee4-a22b-4bde-81f0-899eb8532f49-service-ca\") pod \"23407ee4-a22b-4bde-81f0-899eb8532f49\" (UID: \"23407ee4-a22b-4bde-81f0-899eb8532f49\") " Sep 10 13:18:50.963616 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:18:50.963601 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/23407ee4-a22b-4bde-81f0-899eb8532f49-console-serving-cert\") pod \"23407ee4-a22b-4bde-81f0-899eb8532f49\" (UID: \"23407ee4-a22b-4bde-81f0-899eb8532f49\") " Sep 10 13:18:50.963811 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:18:50.963617 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/23407ee4-a22b-4bde-81f0-899eb8532f49-trusted-ca-bundle\") pod \"23407ee4-a22b-4bde-81f0-899eb8532f49\" (UID: \"23407ee4-a22b-4bde-81f0-899eb8532f49\") " Sep 10 13:18:50.963811 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:18:50.963643 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/23407ee4-a22b-4bde-81f0-899eb8532f49-console-oauth-config\") pod \"23407ee4-a22b-4bde-81f0-899eb8532f49\" (UID: \"23407ee4-a22b-4bde-81f0-899eb8532f49\") " Sep 10 13:18:50.963910 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:18:50.963865 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/23407ee4-a22b-4bde-81f0-899eb8532f49-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "23407ee4-a22b-4bde-81f0-899eb8532f49" (UID: "23407ee4-a22b-4bde-81f0-899eb8532f49"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 10 13:18:50.963910 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:18:50.963888 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/23407ee4-a22b-4bde-81f0-899eb8532f49-console-config" (OuterVolumeSpecName: "console-config") pod "23407ee4-a22b-4bde-81f0-899eb8532f49" (UID: "23407ee4-a22b-4bde-81f0-899eb8532f49"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 10 13:18:50.964122 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:18:50.964087 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/23407ee4-a22b-4bde-81f0-899eb8532f49-service-ca" (OuterVolumeSpecName: "service-ca") pod "23407ee4-a22b-4bde-81f0-899eb8532f49" (UID: "23407ee4-a22b-4bde-81f0-899eb8532f49"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 10 13:18:50.964122 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:18:50.964102 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/23407ee4-a22b-4bde-81f0-899eb8532f49-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "23407ee4-a22b-4bde-81f0-899eb8532f49" (UID: "23407ee4-a22b-4bde-81f0-899eb8532f49"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 10 13:18:50.966015 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:18:50.965991 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/23407ee4-a22b-4bde-81f0-899eb8532f49-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "23407ee4-a22b-4bde-81f0-899eb8532f49" (UID: "23407ee4-a22b-4bde-81f0-899eb8532f49"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 10 13:18:50.966107 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:18:50.966064 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/23407ee4-a22b-4bde-81f0-899eb8532f49-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "23407ee4-a22b-4bde-81f0-899eb8532f49" (UID: "23407ee4-a22b-4bde-81f0-899eb8532f49"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 10 13:18:50.966107 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:18:50.966093 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/23407ee4-a22b-4bde-81f0-899eb8532f49-kube-api-access-75ww7" (OuterVolumeSpecName: "kube-api-access-75ww7") pod "23407ee4-a22b-4bde-81f0-899eb8532f49" (UID: "23407ee4-a22b-4bde-81f0-899eb8532f49"). InnerVolumeSpecName "kube-api-access-75ww7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 10 13:18:51.064894 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:18:51.064864 2557 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/23407ee4-a22b-4bde-81f0-899eb8532f49-console-serving-cert\") on node \"ip-10-0-139-174.ec2.internal\" DevicePath \"\"" Sep 10 13:18:51.064894 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:18:51.064890 2557 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/23407ee4-a22b-4bde-81f0-899eb8532f49-trusted-ca-bundle\") on node \"ip-10-0-139-174.ec2.internal\" DevicePath \"\"" Sep 10 13:18:51.064894 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:18:51.064899 2557 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/23407ee4-a22b-4bde-81f0-899eb8532f49-console-oauth-config\") on node \"ip-10-0-139-174.ec2.internal\" DevicePath \"\"" Sep 10 13:18:51.065109 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:18:51.064907 2557 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/23407ee4-a22b-4bde-81f0-899eb8532f49-oauth-serving-cert\") on node \"ip-10-0-139-174.ec2.internal\" DevicePath \"\"" Sep 10 13:18:51.065109 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:18:51.064916 2557 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/23407ee4-a22b-4bde-81f0-899eb8532f49-console-config\") on node \"ip-10-0-139-174.ec2.internal\" DevicePath \"\"" Sep 10 13:18:51.065109 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:18:51.064924 2557 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-75ww7\" (UniqueName: \"kubernetes.io/projected/23407ee4-a22b-4bde-81f0-899eb8532f49-kube-api-access-75ww7\") on node \"ip-10-0-139-174.ec2.internal\" DevicePath \"\"" Sep 10 13:18:51.065109 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:18:51.064933 2557 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/23407ee4-a22b-4bde-81f0-899eb8532f49-service-ca\") on node \"ip-10-0-139-174.ec2.internal\" DevicePath \"\"" Sep 10 13:18:51.708489 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:18:51.708457 2557 generic.go:356] "Generic (PLEG): container finished" podID="23407ee4-a22b-4bde-81f0-899eb8532f49" containerID="6a429a033af17365bf1758c54ac20a2d8aa9f87db795c42d23d17a808ebd7ea4" exitCode=0 Sep 10 13:18:51.708847 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:18:51.708506 2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7448d46fdc-jj7z4" Sep 10 13:18:51.708847 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:18:51.708506 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7448d46fdc-jj7z4" event={"ID":"23407ee4-a22b-4bde-81f0-899eb8532f49","Type":"ContainerDied","Data":"6a429a033af17365bf1758c54ac20a2d8aa9f87db795c42d23d17a808ebd7ea4"} Sep 10 13:18:51.708847 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:18:51.708632 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7448d46fdc-jj7z4" event={"ID":"23407ee4-a22b-4bde-81f0-899eb8532f49","Type":"ContainerDied","Data":"95e49e1bff384635b55269c587cfa81c0da8ef827ba983f421270434a75eb072"} Sep 10 13:18:51.708847 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:18:51.708661 2557 scope.go:117] "RemoveContainer" containerID="6a429a033af17365bf1758c54ac20a2d8aa9f87db795c42d23d17a808ebd7ea4" Sep 10 13:18:51.717154 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:18:51.717138 2557 scope.go:117] "RemoveContainer" containerID="6a429a033af17365bf1758c54ac20a2d8aa9f87db795c42d23d17a808ebd7ea4" Sep 10 13:18:51.717438 ip-10-0-139-174 kubenswrapper[2557]: E0910 13:18:51.717401 2557 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6a429a033af17365bf1758c54ac20a2d8aa9f87db795c42d23d17a808ebd7ea4\": container with ID starting with 6a429a033af17365bf1758c54ac20a2d8aa9f87db795c42d23d17a808ebd7ea4 not found: ID does not exist" containerID="6a429a033af17365bf1758c54ac20a2d8aa9f87db795c42d23d17a808ebd7ea4" Sep 10 13:18:51.717514 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:18:51.717442 2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6a429a033af17365bf1758c54ac20a2d8aa9f87db795c42d23d17a808ebd7ea4"} err="failed to get container status \"6a429a033af17365bf1758c54ac20a2d8aa9f87db795c42d23d17a808ebd7ea4\": rpc error: code = NotFound desc = could not find container \"6a429a033af17365bf1758c54ac20a2d8aa9f87db795c42d23d17a808ebd7ea4\": container with ID starting with 6a429a033af17365bf1758c54ac20a2d8aa9f87db795c42d23d17a808ebd7ea4 not found: ID does not exist" Sep 10 13:18:51.728874 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:18:51.728853 2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-7448d46fdc-jj7z4"] Sep 10 13:18:51.731149 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:18:51.731130 2557 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-7448d46fdc-jj7z4"] Sep 10 13:18:52.217563 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:18:52.217542 2557 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="23407ee4-a22b-4bde-81f0-899eb8532f49" path="/var/lib/kubelet/pods/23407ee4-a22b-4bde-81f0-899eb8532f49/volumes" Sep 10 13:19:03.644801 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:19:03.644772 2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-6ggtx" Sep 10 13:19:05.595470 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:19:05.595444 2557 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/metrics-server-6b596c7f59-v6hk8" Sep 10 13:19:05.599293 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:19:05.599266 2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/metrics-server-6b596c7f59-v6hk8" Sep 10 13:19:41.587573 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:19:41.587534 2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Sep 10 13:19:41.588361 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:19:41.587967 2557 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="1b94312a-a7f9-4743-8057-a5dae093befb" containerName="alertmanager" containerID="cri-o://9f6ee9b7d6a4567cc2311feb9a9101aabad8634cbcca8e9a8b02025c3a9791af" gracePeriod=120 Sep 10 13:19:41.588361 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:19:41.588029 2557 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="1b94312a-a7f9-4743-8057-a5dae093befb" containerName="kube-rbac-proxy-metric" containerID="cri-o://8a1ba4a366a39a1d0834a8c2111525246b66800b9f8e54e432d75b2c72b4ae66" gracePeriod=120 Sep 10 13:19:41.588361 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:19:41.588067 2557 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="1b94312a-a7f9-4743-8057-a5dae093befb" containerName="kube-rbac-proxy-web" containerID="cri-o://08d07cb07e5418ef818ba260e3db681fd3a3d4427e5d3c4e9de6e447d4c7c2fd" gracePeriod=120 Sep 10 13:19:41.588361 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:19:41.588078 2557 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="1b94312a-a7f9-4743-8057-a5dae093befb" containerName="config-reloader" containerID="cri-o://5bd13798b08874bab211fe5a06d81fda9792013d4277386570bdf8a81e162ae1" gracePeriod=120 Sep 10 13:19:41.588361 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:19:41.588256 2557 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="1b94312a-a7f9-4743-8057-a5dae093befb" containerName="kube-rbac-proxy" containerID="cri-o://4d5c948938735bf7898acbabf4a14a44db170c609d0930457d158212483741f6" gracePeriod=120 Sep 10 13:19:41.588361 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:19:41.588307 2557 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="1b94312a-a7f9-4743-8057-a5dae093befb" containerName="prom-label-proxy" containerID="cri-o://1cf9d39367a41907d3e7988b6a28b15562617005e2909d44160c84d7e5f0733a" gracePeriod=120 Sep 10 13:19:41.890585 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:19:41.890518 2557 generic.go:356] "Generic (PLEG): container finished" podID="1b94312a-a7f9-4743-8057-a5dae093befb" containerID="1cf9d39367a41907d3e7988b6a28b15562617005e2909d44160c84d7e5f0733a" exitCode=0 Sep 10 13:19:41.890585 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:19:41.890543 2557 generic.go:356] "Generic (PLEG): container finished" podID="1b94312a-a7f9-4743-8057-a5dae093befb" containerID="8a1ba4a366a39a1d0834a8c2111525246b66800b9f8e54e432d75b2c72b4ae66" exitCode=0 Sep 10 13:19:41.890585 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:19:41.890550 2557 generic.go:356] "Generic (PLEG): container finished" podID="1b94312a-a7f9-4743-8057-a5dae093befb" containerID="4d5c948938735bf7898acbabf4a14a44db170c609d0930457d158212483741f6" exitCode=0 Sep 10 13:19:41.890585 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:19:41.890556 2557 generic.go:356] "Generic (PLEG): container finished" podID="1b94312a-a7f9-4743-8057-a5dae093befb" containerID="5bd13798b08874bab211fe5a06d81fda9792013d4277386570bdf8a81e162ae1" exitCode=0 Sep 10 13:19:41.890585 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:19:41.890561 2557 generic.go:356] "Generic (PLEG): container finished" podID="1b94312a-a7f9-4743-8057-a5dae093befb" containerID="9f6ee9b7d6a4567cc2311feb9a9101aabad8634cbcca8e9a8b02025c3a9791af" exitCode=0 Sep 10 13:19:41.890877 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:19:41.890592 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"1b94312a-a7f9-4743-8057-a5dae093befb","Type":"ContainerDied","Data":"1cf9d39367a41907d3e7988b6a28b15562617005e2909d44160c84d7e5f0733a"} Sep 10 13:19:41.890877 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:19:41.890671 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"1b94312a-a7f9-4743-8057-a5dae093befb","Type":"ContainerDied","Data":"8a1ba4a366a39a1d0834a8c2111525246b66800b9f8e54e432d75b2c72b4ae66"} Sep 10 13:19:41.890877 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:19:41.890711 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"1b94312a-a7f9-4743-8057-a5dae093befb","Type":"ContainerDied","Data":"4d5c948938735bf7898acbabf4a14a44db170c609d0930457d158212483741f6"} Sep 10 13:19:41.890877 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:19:41.890753 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"1b94312a-a7f9-4743-8057-a5dae093befb","Type":"ContainerDied","Data":"5bd13798b08874bab211fe5a06d81fda9792013d4277386570bdf8a81e162ae1"} Sep 10 13:19:41.890877 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:19:41.890793 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"1b94312a-a7f9-4743-8057-a5dae093befb","Type":"ContainerDied","Data":"9f6ee9b7d6a4567cc2311feb9a9101aabad8634cbcca8e9a8b02025c3a9791af"} Sep 10 13:19:42.826277 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:19:42.826256 2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Sep 10 13:19:42.849165 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:19:42.849138 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/1b94312a-a7f9-4743-8057-a5dae093befb-secret-alertmanager-kube-rbac-proxy-web\") pod \"1b94312a-a7f9-4743-8057-a5dae093befb\" (UID: \"1b94312a-a7f9-4743-8057-a5dae093befb\") " Sep 10 13:19:42.849266 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:19:42.849170 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/1b94312a-a7f9-4743-8057-a5dae093befb-alertmanager-main-db\") pod \"1b94312a-a7f9-4743-8057-a5dae093befb\" (UID: \"1b94312a-a7f9-4743-8057-a5dae093befb\") " Sep 10 13:19:42.849266 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:19:42.849193 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/1b94312a-a7f9-4743-8057-a5dae093befb-web-config\") pod \"1b94312a-a7f9-4743-8057-a5dae093befb\" (UID: \"1b94312a-a7f9-4743-8057-a5dae093befb\") " Sep 10 13:19:42.849266 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:19:42.849211 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/1b94312a-a7f9-4743-8057-a5dae093befb-tls-assets\") pod \"1b94312a-a7f9-4743-8057-a5dae093befb\" (UID: \"1b94312a-a7f9-4743-8057-a5dae093befb\") " Sep 10 13:19:42.849392 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:19:42.849262 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/1b94312a-a7f9-4743-8057-a5dae093befb-secret-alertmanager-kube-rbac-proxy\") pod \"1b94312a-a7f9-4743-8057-a5dae093befb\" (UID: \"1b94312a-a7f9-4743-8057-a5dae093befb\") " Sep 10 13:19:42.849392 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:19:42.849299 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/1b94312a-a7f9-4743-8057-a5dae093befb-metrics-client-ca\") pod \"1b94312a-a7f9-4743-8057-a5dae093befb\" (UID: \"1b94312a-a7f9-4743-8057-a5dae093befb\") " Sep 10 13:19:42.849392 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:19:42.849328 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1b94312a-a7f9-4743-8057-a5dae093befb-alertmanager-trusted-ca-bundle\") pod \"1b94312a-a7f9-4743-8057-a5dae093befb\" (UID: \"1b94312a-a7f9-4743-8057-a5dae093befb\") " Sep 10 13:19:42.849392 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:19:42.849356 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/1b94312a-a7f9-4743-8057-a5dae093befb-secret-alertmanager-main-tls\") pod \"1b94312a-a7f9-4743-8057-a5dae093befb\" (UID: \"1b94312a-a7f9-4743-8057-a5dae093befb\") " Sep 10 13:19:42.849564 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:19:42.849391 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/1b94312a-a7f9-4743-8057-a5dae093befb-cluster-tls-config\") pod \"1b94312a-a7f9-4743-8057-a5dae093befb\" (UID: \"1b94312a-a7f9-4743-8057-a5dae093befb\") " Sep 10 13:19:42.849564 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:19:42.849449 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vddsr\" (UniqueName: \"kubernetes.io/projected/1b94312a-a7f9-4743-8057-a5dae093befb-kube-api-access-vddsr\") pod \"1b94312a-a7f9-4743-8057-a5dae093befb\" (UID: \"1b94312a-a7f9-4743-8057-a5dae093befb\") " Sep 10 13:19:42.849564 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:19:42.849478 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/1b94312a-a7f9-4743-8057-a5dae093befb-secret-alertmanager-kube-rbac-proxy-metric\") pod \"1b94312a-a7f9-4743-8057-a5dae093befb\" (UID: \"1b94312a-a7f9-4743-8057-a5dae093befb\") " Sep 10 13:19:42.849564 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:19:42.849510 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/1b94312a-a7f9-4743-8057-a5dae093befb-config-out\") pod \"1b94312a-a7f9-4743-8057-a5dae093befb\" (UID: \"1b94312a-a7f9-4743-8057-a5dae093befb\") " Sep 10 13:19:42.849564 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:19:42.849544 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/1b94312a-a7f9-4743-8057-a5dae093befb-config-volume\") pod \"1b94312a-a7f9-4743-8057-a5dae093befb\" (UID: \"1b94312a-a7f9-4743-8057-a5dae093befb\") " Sep 10 13:19:42.849564 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:19:42.849554 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1b94312a-a7f9-4743-8057-a5dae093befb-alertmanager-main-db" (OuterVolumeSpecName: "alertmanager-main-db") pod "1b94312a-a7f9-4743-8057-a5dae093befb" (UID: "1b94312a-a7f9-4743-8057-a5dae093befb"). InnerVolumeSpecName "alertmanager-main-db". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 13:19:42.849851 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:19:42.849757 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1b94312a-a7f9-4743-8057-a5dae093befb-alertmanager-trusted-ca-bundle" (OuterVolumeSpecName: "alertmanager-trusted-ca-bundle") pod "1b94312a-a7f9-4743-8057-a5dae093befb" (UID: "1b94312a-a7f9-4743-8057-a5dae093befb"). InnerVolumeSpecName "alertmanager-trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 10 13:19:42.849851 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:19:42.849773 2557 reconciler_common.go:299] "Volume detached for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/1b94312a-a7f9-4743-8057-a5dae093befb-alertmanager-main-db\") on node \"ip-10-0-139-174.ec2.internal\" DevicePath \"\"" Sep 10 13:19:42.850332 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:19:42.850300 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1b94312a-a7f9-4743-8057-a5dae093befb-metrics-client-ca" (OuterVolumeSpecName: "metrics-client-ca") pod "1b94312a-a7f9-4743-8057-a5dae093befb" (UID: "1b94312a-a7f9-4743-8057-a5dae093befb"). InnerVolumeSpecName "metrics-client-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 10 13:19:42.853104 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:19:42.853075 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1b94312a-a7f9-4743-8057-a5dae093befb-secret-alertmanager-kube-rbac-proxy" (OuterVolumeSpecName: "secret-alertmanager-kube-rbac-proxy") pod "1b94312a-a7f9-4743-8057-a5dae093befb" (UID: "1b94312a-a7f9-4743-8057-a5dae093befb"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 10 13:19:42.853235 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:19:42.853198 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1b94312a-a7f9-4743-8057-a5dae093befb-config-volume" (OuterVolumeSpecName: "config-volume") pod "1b94312a-a7f9-4743-8057-a5dae093befb" (UID: "1b94312a-a7f9-4743-8057-a5dae093befb"). InnerVolumeSpecName "config-volume". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 10 13:19:42.853394 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:19:42.853310 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1b94312a-a7f9-4743-8057-a5dae093befb-tls-assets" (OuterVolumeSpecName: "tls-assets") pod "1b94312a-a7f9-4743-8057-a5dae093befb" (UID: "1b94312a-a7f9-4743-8057-a5dae093befb"). InnerVolumeSpecName "tls-assets". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 10 13:19:42.853394 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:19:42.853348 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1b94312a-a7f9-4743-8057-a5dae093befb-secret-alertmanager-main-tls" (OuterVolumeSpecName: "secret-alertmanager-main-tls") pod "1b94312a-a7f9-4743-8057-a5dae093befb" (UID: "1b94312a-a7f9-4743-8057-a5dae093befb"). InnerVolumeSpecName "secret-alertmanager-main-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 10 13:19:42.853582 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:19:42.853561 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1b94312a-a7f9-4743-8057-a5dae093befb-kube-api-access-vddsr" (OuterVolumeSpecName: "kube-api-access-vddsr") pod "1b94312a-a7f9-4743-8057-a5dae093befb" (UID: "1b94312a-a7f9-4743-8057-a5dae093befb"). InnerVolumeSpecName "kube-api-access-vddsr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 10 13:19:42.853646 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:19:42.853573 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1b94312a-a7f9-4743-8057-a5dae093befb-secret-alertmanager-kube-rbac-proxy-web" (OuterVolumeSpecName: "secret-alertmanager-kube-rbac-proxy-web") pod "1b94312a-a7f9-4743-8057-a5dae093befb" (UID: "1b94312a-a7f9-4743-8057-a5dae093befb"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy-web". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 10 13:19:42.854165 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:19:42.854133 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1b94312a-a7f9-4743-8057-a5dae093befb-config-out" (OuterVolumeSpecName: "config-out") pod "1b94312a-a7f9-4743-8057-a5dae093befb" (UID: "1b94312a-a7f9-4743-8057-a5dae093befb"). InnerVolumeSpecName "config-out". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 13:19:42.854464 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:19:42.854401 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1b94312a-a7f9-4743-8057-a5dae093befb-secret-alertmanager-kube-rbac-proxy-metric" (OuterVolumeSpecName: "secret-alertmanager-kube-rbac-proxy-metric") pod "1b94312a-a7f9-4743-8057-a5dae093befb" (UID: "1b94312a-a7f9-4743-8057-a5dae093befb"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy-metric". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 10 13:19:42.857213 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:19:42.857160 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1b94312a-a7f9-4743-8057-a5dae093befb-cluster-tls-config" (OuterVolumeSpecName: "cluster-tls-config") pod "1b94312a-a7f9-4743-8057-a5dae093befb" (UID: "1b94312a-a7f9-4743-8057-a5dae093befb"). InnerVolumeSpecName "cluster-tls-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 10 13:19:42.863247 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:19:42.863227 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1b94312a-a7f9-4743-8057-a5dae093befb-web-config" (OuterVolumeSpecName: "web-config") pod "1b94312a-a7f9-4743-8057-a5dae093befb" (UID: "1b94312a-a7f9-4743-8057-a5dae093befb"). InnerVolumeSpecName "web-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 10 13:19:42.898084 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:19:42.898055 2557 generic.go:356] "Generic (PLEG): container finished" podID="1b94312a-a7f9-4743-8057-a5dae093befb" containerID="08d07cb07e5418ef818ba260e3db681fd3a3d4427e5d3c4e9de6e447d4c7c2fd" exitCode=0 Sep 10 13:19:42.898209 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:19:42.898128 2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Sep 10 13:19:42.898209 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:19:42.898147 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"1b94312a-a7f9-4743-8057-a5dae093befb","Type":"ContainerDied","Data":"08d07cb07e5418ef818ba260e3db681fd3a3d4427e5d3c4e9de6e447d4c7c2fd"} Sep 10 13:19:42.898282 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:19:42.898243 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"1b94312a-a7f9-4743-8057-a5dae093befb","Type":"ContainerDied","Data":"e9dbc24439e0b3ab25f7362e18ce06ef420558bffab84568cd626e5d64c7783f"} Sep 10 13:19:42.898315 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:19:42.898302 2557 scope.go:117] "RemoveContainer" containerID="1cf9d39367a41907d3e7988b6a28b15562617005e2909d44160c84d7e5f0733a" Sep 10 13:19:42.906917 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:19:42.906898 2557 scope.go:117] "RemoveContainer" containerID="8a1ba4a366a39a1d0834a8c2111525246b66800b9f8e54e432d75b2c72b4ae66" Sep 10 13:19:42.914292 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:19:42.914276 2557 scope.go:117] "RemoveContainer" containerID="4d5c948938735bf7898acbabf4a14a44db170c609d0930457d158212483741f6" Sep 10 13:19:42.920853 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:19:42.920827 2557 scope.go:117] "RemoveContainer" containerID="08d07cb07e5418ef818ba260e3db681fd3a3d4427e5d3c4e9de6e447d4c7c2fd" Sep 10 13:19:42.927265 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:19:42.927243 2557 scope.go:117] "RemoveContainer" containerID="5bd13798b08874bab211fe5a06d81fda9792013d4277386570bdf8a81e162ae1" Sep 10 13:19:42.935450 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:19:42.935336 2557 scope.go:117] "RemoveContainer" containerID="9f6ee9b7d6a4567cc2311feb9a9101aabad8634cbcca8e9a8b02025c3a9791af" Sep 10 13:19:42.938549 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:19:42.938529 2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Sep 10 13:19:42.943925 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:19:42.943908 2557 scope.go:117] "RemoveContainer" containerID="f30ac0076533ce6d1bfbf56b70ed2614e8c1cad5bb434e0c296a5a0a0e00a2d1" Sep 10 13:19:42.948193 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:19:42.948170 2557 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Sep 10 13:19:42.950597 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:19:42.950575 2557 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/1b94312a-a7f9-4743-8057-a5dae093befb-secret-alertmanager-kube-rbac-proxy-web\") on node \"ip-10-0-139-174.ec2.internal\" DevicePath \"\"" Sep 10 13:19:42.950597 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:19:42.950595 2557 reconciler_common.go:299] "Volume detached for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/1b94312a-a7f9-4743-8057-a5dae093befb-web-config\") on node \"ip-10-0-139-174.ec2.internal\" DevicePath \"\"" Sep 10 13:19:42.950742 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:19:42.950605 2557 reconciler_common.go:299] "Volume detached for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/1b94312a-a7f9-4743-8057-a5dae093befb-tls-assets\") on node \"ip-10-0-139-174.ec2.internal\" DevicePath \"\"" Sep 10 13:19:42.950742 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:19:42.950613 2557 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/1b94312a-a7f9-4743-8057-a5dae093befb-secret-alertmanager-kube-rbac-proxy\") on node \"ip-10-0-139-174.ec2.internal\" DevicePath \"\"" Sep 10 13:19:42.950742 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:19:42.950622 2557 reconciler_common.go:299] "Volume detached for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/1b94312a-a7f9-4743-8057-a5dae093befb-metrics-client-ca\") on node \"ip-10-0-139-174.ec2.internal\" DevicePath \"\"" Sep 10 13:19:42.950742 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:19:42.950631 2557 reconciler_common.go:299] "Volume detached for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1b94312a-a7f9-4743-8057-a5dae093befb-alertmanager-trusted-ca-bundle\") on node \"ip-10-0-139-174.ec2.internal\" DevicePath \"\"" Sep 10 13:19:42.950742 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:19:42.950640 2557 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/1b94312a-a7f9-4743-8057-a5dae093befb-secret-alertmanager-main-tls\") on node \"ip-10-0-139-174.ec2.internal\" DevicePath \"\"" Sep 10 13:19:42.950742 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:19:42.950648 2557 reconciler_common.go:299] "Volume detached for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/1b94312a-a7f9-4743-8057-a5dae093befb-cluster-tls-config\") on node \"ip-10-0-139-174.ec2.internal\" DevicePath \"\"" Sep 10 13:19:42.950742 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:19:42.950656 2557 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vddsr\" (UniqueName: \"kubernetes.io/projected/1b94312a-a7f9-4743-8057-a5dae093befb-kube-api-access-vddsr\") on node \"ip-10-0-139-174.ec2.internal\" DevicePath \"\"" Sep 10 13:19:42.950742 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:19:42.950664 2557 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/1b94312a-a7f9-4743-8057-a5dae093befb-secret-alertmanager-kube-rbac-proxy-metric\") on node \"ip-10-0-139-174.ec2.internal\" DevicePath \"\"" Sep 10 13:19:42.950742 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:19:42.950673 2557 reconciler_common.go:299] "Volume detached for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/1b94312a-a7f9-4743-8057-a5dae093befb-config-out\") on node \"ip-10-0-139-174.ec2.internal\" DevicePath \"\"" Sep 10 13:19:42.950742 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:19:42.950682 2557 reconciler_common.go:299] "Volume detached for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/1b94312a-a7f9-4743-8057-a5dae093befb-config-volume\") on node \"ip-10-0-139-174.ec2.internal\" DevicePath \"\"" Sep 10 13:19:42.951164 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:19:42.951153 2557 scope.go:117] "RemoveContainer" containerID="1cf9d39367a41907d3e7988b6a28b15562617005e2909d44160c84d7e5f0733a" Sep 10 13:19:42.951413 ip-10-0-139-174 kubenswrapper[2557]: E0910 13:19:42.951396 2557 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1cf9d39367a41907d3e7988b6a28b15562617005e2909d44160c84d7e5f0733a\": container with ID starting with 1cf9d39367a41907d3e7988b6a28b15562617005e2909d44160c84d7e5f0733a not found: ID does not exist" containerID="1cf9d39367a41907d3e7988b6a28b15562617005e2909d44160c84d7e5f0733a" Sep 10 13:19:42.951503 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:19:42.951435 2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1cf9d39367a41907d3e7988b6a28b15562617005e2909d44160c84d7e5f0733a"} err="failed to get container status \"1cf9d39367a41907d3e7988b6a28b15562617005e2909d44160c84d7e5f0733a\": rpc error: code = NotFound desc = could not find container \"1cf9d39367a41907d3e7988b6a28b15562617005e2909d44160c84d7e5f0733a\": container with ID starting with 1cf9d39367a41907d3e7988b6a28b15562617005e2909d44160c84d7e5f0733a not found: ID does not exist" Sep 10 13:19:42.951503 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:19:42.951459 2557 scope.go:117] "RemoveContainer" containerID="8a1ba4a366a39a1d0834a8c2111525246b66800b9f8e54e432d75b2c72b4ae66" Sep 10 13:19:42.951738 ip-10-0-139-174 kubenswrapper[2557]: E0910 13:19:42.951721 2557 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8a1ba4a366a39a1d0834a8c2111525246b66800b9f8e54e432d75b2c72b4ae66\": container with ID starting with 8a1ba4a366a39a1d0834a8c2111525246b66800b9f8e54e432d75b2c72b4ae66 not found: ID does not exist" containerID="8a1ba4a366a39a1d0834a8c2111525246b66800b9f8e54e432d75b2c72b4ae66" Sep 10 13:19:42.951781 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:19:42.951753 2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8a1ba4a366a39a1d0834a8c2111525246b66800b9f8e54e432d75b2c72b4ae66"} err="failed to get container status \"8a1ba4a366a39a1d0834a8c2111525246b66800b9f8e54e432d75b2c72b4ae66\": rpc error: code = NotFound desc = could not find container \"8a1ba4a366a39a1d0834a8c2111525246b66800b9f8e54e432d75b2c72b4ae66\": container with ID starting with 8a1ba4a366a39a1d0834a8c2111525246b66800b9f8e54e432d75b2c72b4ae66 not found: ID does not exist" Sep 10 13:19:42.951781 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:19:42.951770 2557 scope.go:117] "RemoveContainer" containerID="4d5c948938735bf7898acbabf4a14a44db170c609d0930457d158212483741f6" Sep 10 13:19:42.952031 ip-10-0-139-174 kubenswrapper[2557]: E0910 13:19:42.952014 2557 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4d5c948938735bf7898acbabf4a14a44db170c609d0930457d158212483741f6\": container with ID starting with 4d5c948938735bf7898acbabf4a14a44db170c609d0930457d158212483741f6 not found: ID does not exist" containerID="4d5c948938735bf7898acbabf4a14a44db170c609d0930457d158212483741f6" Sep 10 13:19:42.952078 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:19:42.952034 2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4d5c948938735bf7898acbabf4a14a44db170c609d0930457d158212483741f6"} err="failed to get container status \"4d5c948938735bf7898acbabf4a14a44db170c609d0930457d158212483741f6\": rpc error: code = NotFound desc = could not find container \"4d5c948938735bf7898acbabf4a14a44db170c609d0930457d158212483741f6\": container with ID starting with 4d5c948938735bf7898acbabf4a14a44db170c609d0930457d158212483741f6 not found: ID does not exist" Sep 10 13:19:42.952078 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:19:42.952049 2557 scope.go:117] "RemoveContainer" containerID="08d07cb07e5418ef818ba260e3db681fd3a3d4427e5d3c4e9de6e447d4c7c2fd" Sep 10 13:19:42.952283 ip-10-0-139-174 kubenswrapper[2557]: E0910 13:19:42.952268 2557 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"08d07cb07e5418ef818ba260e3db681fd3a3d4427e5d3c4e9de6e447d4c7c2fd\": container with ID starting with 08d07cb07e5418ef818ba260e3db681fd3a3d4427e5d3c4e9de6e447d4c7c2fd not found: ID does not exist" containerID="08d07cb07e5418ef818ba260e3db681fd3a3d4427e5d3c4e9de6e447d4c7c2fd" Sep 10 13:19:42.952325 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:19:42.952287 2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"08d07cb07e5418ef818ba260e3db681fd3a3d4427e5d3c4e9de6e447d4c7c2fd"} err="failed to get container status \"08d07cb07e5418ef818ba260e3db681fd3a3d4427e5d3c4e9de6e447d4c7c2fd\": rpc error: code = NotFound desc = could not find container \"08d07cb07e5418ef818ba260e3db681fd3a3d4427e5d3c4e9de6e447d4c7c2fd\": container with ID starting with 08d07cb07e5418ef818ba260e3db681fd3a3d4427e5d3c4e9de6e447d4c7c2fd not found: ID does not exist" Sep 10 13:19:42.952325 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:19:42.952302 2557 scope.go:117] "RemoveContainer" containerID="5bd13798b08874bab211fe5a06d81fda9792013d4277386570bdf8a81e162ae1" Sep 10 13:19:42.952796 ip-10-0-139-174 kubenswrapper[2557]: E0910 13:19:42.952779 2557 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5bd13798b08874bab211fe5a06d81fda9792013d4277386570bdf8a81e162ae1\": container with ID starting with 5bd13798b08874bab211fe5a06d81fda9792013d4277386570bdf8a81e162ae1 not found: ID does not exist" containerID="5bd13798b08874bab211fe5a06d81fda9792013d4277386570bdf8a81e162ae1" Sep 10 13:19:42.952860 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:19:42.952799 2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5bd13798b08874bab211fe5a06d81fda9792013d4277386570bdf8a81e162ae1"} err="failed to get container status \"5bd13798b08874bab211fe5a06d81fda9792013d4277386570bdf8a81e162ae1\": rpc error: code = NotFound desc = could not find container \"5bd13798b08874bab211fe5a06d81fda9792013d4277386570bdf8a81e162ae1\": container with ID starting with 5bd13798b08874bab211fe5a06d81fda9792013d4277386570bdf8a81e162ae1 not found: ID does not exist" Sep 10 13:19:42.952860 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:19:42.952812 2557 scope.go:117] "RemoveContainer" containerID="9f6ee9b7d6a4567cc2311feb9a9101aabad8634cbcca8e9a8b02025c3a9791af" Sep 10 13:19:42.953099 ip-10-0-139-174 kubenswrapper[2557]: E0910 13:19:42.953084 2557 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9f6ee9b7d6a4567cc2311feb9a9101aabad8634cbcca8e9a8b02025c3a9791af\": container with ID starting with 9f6ee9b7d6a4567cc2311feb9a9101aabad8634cbcca8e9a8b02025c3a9791af not found: ID does not exist" containerID="9f6ee9b7d6a4567cc2311feb9a9101aabad8634cbcca8e9a8b02025c3a9791af" Sep 10 13:19:42.953140 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:19:42.953103 2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9f6ee9b7d6a4567cc2311feb9a9101aabad8634cbcca8e9a8b02025c3a9791af"} err="failed to get container status \"9f6ee9b7d6a4567cc2311feb9a9101aabad8634cbcca8e9a8b02025c3a9791af\": rpc error: code = NotFound desc = could not find container \"9f6ee9b7d6a4567cc2311feb9a9101aabad8634cbcca8e9a8b02025c3a9791af\": container with ID starting with 9f6ee9b7d6a4567cc2311feb9a9101aabad8634cbcca8e9a8b02025c3a9791af not found: ID does not exist" Sep 10 13:19:42.953140 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:19:42.953116 2557 scope.go:117] "RemoveContainer" containerID="f30ac0076533ce6d1bfbf56b70ed2614e8c1cad5bb434e0c296a5a0a0e00a2d1" Sep 10 13:19:42.953363 ip-10-0-139-174 kubenswrapper[2557]: E0910 13:19:42.953347 2557 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f30ac0076533ce6d1bfbf56b70ed2614e8c1cad5bb434e0c296a5a0a0e00a2d1\": container with ID starting with f30ac0076533ce6d1bfbf56b70ed2614e8c1cad5bb434e0c296a5a0a0e00a2d1 not found: ID does not exist" containerID="f30ac0076533ce6d1bfbf56b70ed2614e8c1cad5bb434e0c296a5a0a0e00a2d1" Sep 10 13:19:42.953401 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:19:42.953365 2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f30ac0076533ce6d1bfbf56b70ed2614e8c1cad5bb434e0c296a5a0a0e00a2d1"} err="failed to get container status \"f30ac0076533ce6d1bfbf56b70ed2614e8c1cad5bb434e0c296a5a0a0e00a2d1\": rpc error: code = NotFound desc = could not find container \"f30ac0076533ce6d1bfbf56b70ed2614e8c1cad5bb434e0c296a5a0a0e00a2d1\": container with ID starting with f30ac0076533ce6d1bfbf56b70ed2614e8c1cad5bb434e0c296a5a0a0e00a2d1 not found: ID does not exist" Sep 10 13:19:43.037079 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:19:43.037054 2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Sep 10 13:19:43.037392 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:19:43.037374 2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1b94312a-a7f9-4743-8057-a5dae093befb" containerName="kube-rbac-proxy" Sep 10 13:19:43.037480 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:19:43.037396 2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="1b94312a-a7f9-4743-8057-a5dae093befb" containerName="kube-rbac-proxy" Sep 10 13:19:43.037480 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:19:43.037411 2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1b94312a-a7f9-4743-8057-a5dae093befb" containerName="kube-rbac-proxy-metric" Sep 10 13:19:43.037480 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:19:43.037437 2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="1b94312a-a7f9-4743-8057-a5dae093befb" containerName="kube-rbac-proxy-metric" Sep 10 13:19:43.037480 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:19:43.037449 2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1b94312a-a7f9-4743-8057-a5dae093befb" containerName="config-reloader" Sep 10 13:19:43.037480 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:19:43.037457 2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="1b94312a-a7f9-4743-8057-a5dae093befb" containerName="config-reloader" Sep 10 13:19:43.037480 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:19:43.037466 2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1b94312a-a7f9-4743-8057-a5dae093befb" containerName="kube-rbac-proxy-web" Sep 10 13:19:43.037480 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:19:43.037474 2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="1b94312a-a7f9-4743-8057-a5dae093befb" containerName="kube-rbac-proxy-web" Sep 10 13:19:43.037679 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:19:43.037509 2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="23407ee4-a22b-4bde-81f0-899eb8532f49" containerName="console" Sep 10 13:19:43.037679 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:19:43.037517 2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="23407ee4-a22b-4bde-81f0-899eb8532f49" containerName="console" Sep 10 13:19:43.037679 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:19:43.037528 2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1b94312a-a7f9-4743-8057-a5dae093befb" containerName="prom-label-proxy" Sep 10 13:19:43.037679 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:19:43.037535 2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="1b94312a-a7f9-4743-8057-a5dae093befb" containerName="prom-label-proxy" Sep 10 13:19:43.037679 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:19:43.037544 2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1b94312a-a7f9-4743-8057-a5dae093befb" containerName="init-config-reloader" Sep 10 13:19:43.037679 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:19:43.037551 2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="1b94312a-a7f9-4743-8057-a5dae093befb" containerName="init-config-reloader" Sep 10 13:19:43.037679 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:19:43.037568 2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1b94312a-a7f9-4743-8057-a5dae093befb" containerName="alertmanager" Sep 10 13:19:43.037679 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:19:43.037576 2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="1b94312a-a7f9-4743-8057-a5dae093befb" containerName="alertmanager" Sep 10 13:19:43.037679 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:19:43.037643 2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="1b94312a-a7f9-4743-8057-a5dae093befb" containerName="config-reloader" Sep 10 13:19:43.037679 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:19:43.037655 2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="23407ee4-a22b-4bde-81f0-899eb8532f49" containerName="console" Sep 10 13:19:43.037679 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:19:43.037664 2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="1b94312a-a7f9-4743-8057-a5dae093befb" containerName="prom-label-proxy" Sep 10 13:19:43.037679 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:19:43.037672 2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="1b94312a-a7f9-4743-8057-a5dae093befb" containerName="kube-rbac-proxy-web" Sep 10 13:19:43.037679 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:19:43.037679 2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="1b94312a-a7f9-4743-8057-a5dae093befb" containerName="kube-rbac-proxy" Sep 10 13:19:43.038019 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:19:43.037688 2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="1b94312a-a7f9-4743-8057-a5dae093befb" containerName="kube-rbac-proxy-metric" Sep 10 13:19:43.038019 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:19:43.037695 2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="1b94312a-a7f9-4743-8057-a5dae093befb" containerName="alertmanager" Sep 10 13:19:43.041343 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:19:43.041327 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Sep 10 13:19:43.043660 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:19:43.043636 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\"" Sep 10 13:19:43.043872 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:19:43.043812 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\"" Sep 10 13:19:43.043967 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:19:43.043877 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\"" Sep 10 13:19:43.043967 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:19:43.043944 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-r67l2\"" Sep 10 13:19:43.043967 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:19:43.043949 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\"" Sep 10 13:19:43.044108 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:19:43.043944 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\"" Sep 10 13:19:43.044208 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:19:43.044194 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\"" Sep 10 13:19:43.044270 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:19:43.044257 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\"" Sep 10 13:19:43.044654 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:19:43.044637 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\"" Sep 10 13:19:43.050599 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:19:43.050572 2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\"" Sep 10 13:19:43.050915 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:19:43.050897 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/c07cb7c8-c14f-44ca-8b96-118bdcef762f-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"c07cb7c8-c14f-44ca-8b96-118bdcef762f\") " pod="openshift-monitoring/alertmanager-main-0" Sep 10 13:19:43.050994 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:19:43.050919 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c07cb7c8-c14f-44ca-8b96-118bdcef762f-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"c07cb7c8-c14f-44ca-8b96-118bdcef762f\") " pod="openshift-monitoring/alertmanager-main-0" Sep 10 13:19:43.050994 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:19:43.050940 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/c07cb7c8-c14f-44ca-8b96-118bdcef762f-web-config\") pod \"alertmanager-main-0\" (UID: \"c07cb7c8-c14f-44ca-8b96-118bdcef762f\") " pod="openshift-monitoring/alertmanager-main-0" Sep 10 13:19:43.050994 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:19:43.050955 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/c07cb7c8-c14f-44ca-8b96-118bdcef762f-config-out\") pod \"alertmanager-main-0\" (UID: \"c07cb7c8-c14f-44ca-8b96-118bdcef762f\") " pod="openshift-monitoring/alertmanager-main-0" Sep 10 13:19:43.050994 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:19:43.050972 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/c07cb7c8-c14f-44ca-8b96-118bdcef762f-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"c07cb7c8-c14f-44ca-8b96-118bdcef762f\") " pod="openshift-monitoring/alertmanager-main-0" Sep 10 13:19:43.050994 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:19:43.050989 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/c07cb7c8-c14f-44ca-8b96-118bdcef762f-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"c07cb7c8-c14f-44ca-8b96-118bdcef762f\") " pod="openshift-monitoring/alertmanager-main-0" Sep 10 13:19:43.051214 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:19:43.051029 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k2kkh\" (UniqueName: \"kubernetes.io/projected/c07cb7c8-c14f-44ca-8b96-118bdcef762f-kube-api-access-k2kkh\") pod \"alertmanager-main-0\" (UID: \"c07cb7c8-c14f-44ca-8b96-118bdcef762f\") " pod="openshift-monitoring/alertmanager-main-0" Sep 10 13:19:43.051214 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:19:43.051083 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/c07cb7c8-c14f-44ca-8b96-118bdcef762f-config-volume\") pod \"alertmanager-main-0\" (UID: \"c07cb7c8-c14f-44ca-8b96-118bdcef762f\") " pod="openshift-monitoring/alertmanager-main-0" Sep 10 13:19:43.051214 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:19:43.051104 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/c07cb7c8-c14f-44ca-8b96-118bdcef762f-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"c07cb7c8-c14f-44ca-8b96-118bdcef762f\") " pod="openshift-monitoring/alertmanager-main-0" Sep 10 13:19:43.051214 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:19:43.051176 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/c07cb7c8-c14f-44ca-8b96-118bdcef762f-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"c07cb7c8-c14f-44ca-8b96-118bdcef762f\") " pod="openshift-monitoring/alertmanager-main-0" Sep 10 13:19:43.051386 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:19:43.051213 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/c07cb7c8-c14f-44ca-8b96-118bdcef762f-tls-assets\") pod \"alertmanager-main-0\" (UID: \"c07cb7c8-c14f-44ca-8b96-118bdcef762f\") " pod="openshift-monitoring/alertmanager-main-0" Sep 10 13:19:43.051386 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:19:43.051237 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/c07cb7c8-c14f-44ca-8b96-118bdcef762f-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"c07cb7c8-c14f-44ca-8b96-118bdcef762f\") " pod="openshift-monitoring/alertmanager-main-0" Sep 10 13:19:43.051386 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:19:43.051281 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/c07cb7c8-c14f-44ca-8b96-118bdcef762f-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"c07cb7c8-c14f-44ca-8b96-118bdcef762f\") " pod="openshift-monitoring/alertmanager-main-0" Sep 10 13:19:43.069772 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:19:43.069751 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Sep 10 13:19:43.151868 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:19:43.151800 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/c07cb7c8-c14f-44ca-8b96-118bdcef762f-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"c07cb7c8-c14f-44ca-8b96-118bdcef762f\") " pod="openshift-monitoring/alertmanager-main-0" Sep 10 13:19:43.151868 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:19:43.151834 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/c07cb7c8-c14f-44ca-8b96-118bdcef762f-tls-assets\") pod \"alertmanager-main-0\" (UID: \"c07cb7c8-c14f-44ca-8b96-118bdcef762f\") " pod="openshift-monitoring/alertmanager-main-0" Sep 10 13:19:43.151868 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:19:43.151852 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/c07cb7c8-c14f-44ca-8b96-118bdcef762f-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"c07cb7c8-c14f-44ca-8b96-118bdcef762f\") " pod="openshift-monitoring/alertmanager-main-0" Sep 10 13:19:43.152027 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:19:43.151883 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/c07cb7c8-c14f-44ca-8b96-118bdcef762f-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"c07cb7c8-c14f-44ca-8b96-118bdcef762f\") " pod="openshift-monitoring/alertmanager-main-0" Sep 10 13:19:43.152083 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:19:43.152061 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/c07cb7c8-c14f-44ca-8b96-118bdcef762f-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"c07cb7c8-c14f-44ca-8b96-118bdcef762f\") " pod="openshift-monitoring/alertmanager-main-0" Sep 10 13:19:43.152131 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:19:43.152101 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c07cb7c8-c14f-44ca-8b96-118bdcef762f-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"c07cb7c8-c14f-44ca-8b96-118bdcef762f\") " pod="openshift-monitoring/alertmanager-main-0" Sep 10 13:19:43.152198 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:19:43.152135 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/c07cb7c8-c14f-44ca-8b96-118bdcef762f-web-config\") pod \"alertmanager-main-0\" (UID: \"c07cb7c8-c14f-44ca-8b96-118bdcef762f\") " pod="openshift-monitoring/alertmanager-main-0" Sep 10 13:19:43.152198 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:19:43.152161 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/c07cb7c8-c14f-44ca-8b96-118bdcef762f-config-out\") pod \"alertmanager-main-0\" (UID: \"c07cb7c8-c14f-44ca-8b96-118bdcef762f\") " pod="openshift-monitoring/alertmanager-main-0" Sep 10 13:19:43.152198 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:19:43.152188 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/c07cb7c8-c14f-44ca-8b96-118bdcef762f-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"c07cb7c8-c14f-44ca-8b96-118bdcef762f\") " pod="openshift-monitoring/alertmanager-main-0" Sep 10 13:19:43.152345 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:19:43.152217 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/c07cb7c8-c14f-44ca-8b96-118bdcef762f-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"c07cb7c8-c14f-44ca-8b96-118bdcef762f\") " pod="openshift-monitoring/alertmanager-main-0" Sep 10 13:19:43.152345 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:19:43.152241 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k2kkh\" (UniqueName: \"kubernetes.io/projected/c07cb7c8-c14f-44ca-8b96-118bdcef762f-kube-api-access-k2kkh\") pod \"alertmanager-main-0\" (UID: \"c07cb7c8-c14f-44ca-8b96-118bdcef762f\") " pod="openshift-monitoring/alertmanager-main-0" Sep 10 13:19:43.152345 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:19:43.152303 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/c07cb7c8-c14f-44ca-8b96-118bdcef762f-config-volume\") pod \"alertmanager-main-0\" (UID: \"c07cb7c8-c14f-44ca-8b96-118bdcef762f\") " pod="openshift-monitoring/alertmanager-main-0" Sep 10 13:19:43.152345 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:19:43.152333 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/c07cb7c8-c14f-44ca-8b96-118bdcef762f-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"c07cb7c8-c14f-44ca-8b96-118bdcef762f\") " pod="openshift-monitoring/alertmanager-main-0" Sep 10 13:19:43.153350 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:19:43.152238 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/c07cb7c8-c14f-44ca-8b96-118bdcef762f-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"c07cb7c8-c14f-44ca-8b96-118bdcef762f\") " pod="openshift-monitoring/alertmanager-main-0" Sep 10 13:19:43.153350 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:19:43.153202 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/c07cb7c8-c14f-44ca-8b96-118bdcef762f-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"c07cb7c8-c14f-44ca-8b96-118bdcef762f\") " pod="openshift-monitoring/alertmanager-main-0" Sep 10 13:19:43.153350 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:19:43.153295 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c07cb7c8-c14f-44ca-8b96-118bdcef762f-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"c07cb7c8-c14f-44ca-8b96-118bdcef762f\") " pod="openshift-monitoring/alertmanager-main-0" Sep 10 13:19:43.155169 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:19:43.155140 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/c07cb7c8-c14f-44ca-8b96-118bdcef762f-tls-assets\") pod \"alertmanager-main-0\" (UID: \"c07cb7c8-c14f-44ca-8b96-118bdcef762f\") " pod="openshift-monitoring/alertmanager-main-0" Sep 10 13:19:43.155264 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:19:43.155226 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/c07cb7c8-c14f-44ca-8b96-118bdcef762f-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"c07cb7c8-c14f-44ca-8b96-118bdcef762f\") " pod="openshift-monitoring/alertmanager-main-0" Sep 10 13:19:43.155345 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:19:43.155322 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/c07cb7c8-c14f-44ca-8b96-118bdcef762f-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"c07cb7c8-c14f-44ca-8b96-118bdcef762f\") " pod="openshift-monitoring/alertmanager-main-0" Sep 10 13:19:43.155723 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:19:43.155691 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/c07cb7c8-c14f-44ca-8b96-118bdcef762f-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"c07cb7c8-c14f-44ca-8b96-118bdcef762f\") " pod="openshift-monitoring/alertmanager-main-0" Sep 10 13:19:43.155723 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:19:43.155712 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/c07cb7c8-c14f-44ca-8b96-118bdcef762f-config-out\") pod \"alertmanager-main-0\" (UID: \"c07cb7c8-c14f-44ca-8b96-118bdcef762f\") " pod="openshift-monitoring/alertmanager-main-0" Sep 10 13:19:43.155902 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:19:43.155878 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/c07cb7c8-c14f-44ca-8b96-118bdcef762f-web-config\") pod \"alertmanager-main-0\" (UID: \"c07cb7c8-c14f-44ca-8b96-118bdcef762f\") " pod="openshift-monitoring/alertmanager-main-0" Sep 10 13:19:43.156017 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:19:43.156004 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/c07cb7c8-c14f-44ca-8b96-118bdcef762f-config-volume\") pod \"alertmanager-main-0\" (UID: \"c07cb7c8-c14f-44ca-8b96-118bdcef762f\") " pod="openshift-monitoring/alertmanager-main-0" Sep 10 13:19:43.156140 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:19:43.156103 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/c07cb7c8-c14f-44ca-8b96-118bdcef762f-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"c07cb7c8-c14f-44ca-8b96-118bdcef762f\") " pod="openshift-monitoring/alertmanager-main-0" Sep 10 13:19:43.157147 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:19:43.157127 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/c07cb7c8-c14f-44ca-8b96-118bdcef762f-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"c07cb7c8-c14f-44ca-8b96-118bdcef762f\") " pod="openshift-monitoring/alertmanager-main-0" Sep 10 13:19:43.164125 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:19:43.164107 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-k2kkh\" (UniqueName: \"kubernetes.io/projected/c07cb7c8-c14f-44ca-8b96-118bdcef762f-kube-api-access-k2kkh\") pod \"alertmanager-main-0\" (UID: \"c07cb7c8-c14f-44ca-8b96-118bdcef762f\") " pod="openshift-monitoring/alertmanager-main-0" Sep 10 13:19:43.351000 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:19:43.350973 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Sep 10 13:19:43.643831 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:19:43.643801 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Sep 10 13:19:43.644953 ip-10-0-139-174 kubenswrapper[2557]: W0910 13:19:43.644925 2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc07cb7c8_c14f_44ca_8b96_118bdcef762f.slice/crio-846b26a1f109067965634124664e3cb65ac41df0010fc8be75425ff1d795ade9 WatchSource:0}: Error finding container 846b26a1f109067965634124664e3cb65ac41df0010fc8be75425ff1d795ade9: Status 404 returned error can't find the container with id 846b26a1f109067965634124664e3cb65ac41df0010fc8be75425ff1d795ade9 Sep 10 13:19:43.903719 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:19:43.903635 2557 generic.go:356] "Generic (PLEG): container finished" podID="c07cb7c8-c14f-44ca-8b96-118bdcef762f" containerID="681a14936ed6a388cc9692cc8594dbff740df526b649b18ee9440ed04af54aa0" exitCode=0 Sep 10 13:19:43.904095 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:19:43.903738 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"c07cb7c8-c14f-44ca-8b96-118bdcef762f","Type":"ContainerDied","Data":"681a14936ed6a388cc9692cc8594dbff740df526b649b18ee9440ed04af54aa0"} Sep 10 13:19:43.904095 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:19:43.903821 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"c07cb7c8-c14f-44ca-8b96-118bdcef762f","Type":"ContainerStarted","Data":"846b26a1f109067965634124664e3cb65ac41df0010fc8be75425ff1d795ade9"} Sep 10 13:19:44.219867 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:19:44.219842 2557 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1b94312a-a7f9-4743-8057-a5dae093befb" path="/var/lib/kubelet/pods/1b94312a-a7f9-4743-8057-a5dae093befb/volumes" Sep 10 13:19:44.911753 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:19:44.911718 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"c07cb7c8-c14f-44ca-8b96-118bdcef762f","Type":"ContainerStarted","Data":"6aa28038b9dcb0799ca2a1ab5c5a6829a50344f82a0d7d5ac803bfaa658e26d1"} Sep 10 13:19:44.912187 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:19:44.911792 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"c07cb7c8-c14f-44ca-8b96-118bdcef762f","Type":"ContainerStarted","Data":"30ec74b75ed847ebe6553188d783c28036c04d421fdb0e83e5316ca9978e6548"} Sep 10 13:19:44.912187 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:19:44.911843 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"c07cb7c8-c14f-44ca-8b96-118bdcef762f","Type":"ContainerStarted","Data":"2103fbff47ba7dfad0aaa2633b7c16d0b1e22876f1ef29cced350b6fba447c4c"} Sep 10 13:19:44.912187 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:19:44.911886 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"c07cb7c8-c14f-44ca-8b96-118bdcef762f","Type":"ContainerStarted","Data":"70ccaf0a08658ba035cc956c700a4e21d78fcb9f096159e51c8dc60876e540ab"} Sep 10 13:19:44.912187 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:19:44.911932 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"c07cb7c8-c14f-44ca-8b96-118bdcef762f","Type":"ContainerStarted","Data":"ec25ba6b71676ba89089f071152c0e2200613cbc7ecb529f203205429d064800"} Sep 10 13:19:44.912187 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:19:44.911987 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"c07cb7c8-c14f-44ca-8b96-118bdcef762f","Type":"ContainerStarted","Data":"02334eed0c6812ecfb1f6ce064c4d28c01c1e3280e23dfb285d31c1146543d99"} Sep 10 13:19:44.958364 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:19:44.958319 2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/alertmanager-main-0" podStartSLOduration=2.942783137 podStartE2EDuration="2.958303904s" totalImagesPullingTime="0s" totalInitContainerRuntime="15.520767ms" isStatefulPod=false podCreationTimestamp="2026-09-10 13:19:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 13:19:44.9578709 +0000 UTC m=+145.264865314" watchObservedRunningTime="2026-09-10 13:19:44.958303904 +0000 UTC m=+145.265298318" Sep 10 13:19:59.531105 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:19:59.531069 2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-6868d857cb-kcx5x"] Sep 10 13:19:59.534310 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:19:59.534283 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6868d857cb-kcx5x" Sep 10 13:19:59.569715 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:19:59.569678 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-6868d857cb-kcx5x"] Sep 10 13:19:59.580609 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:19:59.580571 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e77492b2-9db5-4716-a2b4-78fcf9011686-trusted-ca-bundle\") pod \"console-6868d857cb-kcx5x\" (UID: \"e77492b2-9db5-4716-a2b4-78fcf9011686\") " pod="openshift-console/console-6868d857cb-kcx5x" Sep 10 13:19:59.580609 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:19:59.580611 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/e77492b2-9db5-4716-a2b4-78fcf9011686-console-serving-cert\") pod \"console-6868d857cb-kcx5x\" (UID: \"e77492b2-9db5-4716-a2b4-78fcf9011686\") " pod="openshift-console/console-6868d857cb-kcx5x" Sep 10 13:19:59.580822 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:19:59.580630 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/e77492b2-9db5-4716-a2b4-78fcf9011686-oauth-serving-cert\") pod \"console-6868d857cb-kcx5x\" (UID: \"e77492b2-9db5-4716-a2b4-78fcf9011686\") " pod="openshift-console/console-6868d857cb-kcx5x" Sep 10 13:19:59.580822 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:19:59.580705 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/e77492b2-9db5-4716-a2b4-78fcf9011686-service-ca\") pod \"console-6868d857cb-kcx5x\" (UID: \"e77492b2-9db5-4716-a2b4-78fcf9011686\") " pod="openshift-console/console-6868d857cb-kcx5x" Sep 10 13:19:59.580822 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:19:59.580763 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vbpdx\" (UniqueName: \"kubernetes.io/projected/e77492b2-9db5-4716-a2b4-78fcf9011686-kube-api-access-vbpdx\") pod \"console-6868d857cb-kcx5x\" (UID: \"e77492b2-9db5-4716-a2b4-78fcf9011686\") " pod="openshift-console/console-6868d857cb-kcx5x" Sep 10 13:19:59.580822 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:19:59.580795 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/e77492b2-9db5-4716-a2b4-78fcf9011686-console-config\") pod \"console-6868d857cb-kcx5x\" (UID: \"e77492b2-9db5-4716-a2b4-78fcf9011686\") " pod="openshift-console/console-6868d857cb-kcx5x" Sep 10 13:19:59.580822 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:19:59.580820 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/e77492b2-9db5-4716-a2b4-78fcf9011686-console-oauth-config\") pod \"console-6868d857cb-kcx5x\" (UID: \"e77492b2-9db5-4716-a2b4-78fcf9011686\") " pod="openshift-console/console-6868d857cb-kcx5x" Sep 10 13:19:59.681900 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:19:59.681855 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/e77492b2-9db5-4716-a2b4-78fcf9011686-console-config\") pod \"console-6868d857cb-kcx5x\" (UID: \"e77492b2-9db5-4716-a2b4-78fcf9011686\") " pod="openshift-console/console-6868d857cb-kcx5x" Sep 10 13:19:59.681900 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:19:59.681902 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/e77492b2-9db5-4716-a2b4-78fcf9011686-console-oauth-config\") pod \"console-6868d857cb-kcx5x\" (UID: \"e77492b2-9db5-4716-a2b4-78fcf9011686\") " pod="openshift-console/console-6868d857cb-kcx5x" Sep 10 13:19:59.682129 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:19:59.682030 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e77492b2-9db5-4716-a2b4-78fcf9011686-trusted-ca-bundle\") pod \"console-6868d857cb-kcx5x\" (UID: \"e77492b2-9db5-4716-a2b4-78fcf9011686\") " pod="openshift-console/console-6868d857cb-kcx5x" Sep 10 13:19:59.682129 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:19:59.682081 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/e77492b2-9db5-4716-a2b4-78fcf9011686-console-serving-cert\") pod \"console-6868d857cb-kcx5x\" (UID: \"e77492b2-9db5-4716-a2b4-78fcf9011686\") " pod="openshift-console/console-6868d857cb-kcx5x" Sep 10 13:19:59.682129 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:19:59.682103 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/e77492b2-9db5-4716-a2b4-78fcf9011686-oauth-serving-cert\") pod \"console-6868d857cb-kcx5x\" (UID: \"e77492b2-9db5-4716-a2b4-78fcf9011686\") " pod="openshift-console/console-6868d857cb-kcx5x" Sep 10 13:19:59.682303 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:19:59.682152 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/e77492b2-9db5-4716-a2b4-78fcf9011686-service-ca\") pod \"console-6868d857cb-kcx5x\" (UID: \"e77492b2-9db5-4716-a2b4-78fcf9011686\") " pod="openshift-console/console-6868d857cb-kcx5x" Sep 10 13:19:59.682303 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:19:59.682190 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vbpdx\" (UniqueName: \"kubernetes.io/projected/e77492b2-9db5-4716-a2b4-78fcf9011686-kube-api-access-vbpdx\") pod \"console-6868d857cb-kcx5x\" (UID: \"e77492b2-9db5-4716-a2b4-78fcf9011686\") " pod="openshift-console/console-6868d857cb-kcx5x" Sep 10 13:19:59.682778 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:19:59.682744 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/e77492b2-9db5-4716-a2b4-78fcf9011686-console-config\") pod \"console-6868d857cb-kcx5x\" (UID: \"e77492b2-9db5-4716-a2b4-78fcf9011686\") " pod="openshift-console/console-6868d857cb-kcx5x" Sep 10 13:19:59.682890 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:19:59.682814 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/e77492b2-9db5-4716-a2b4-78fcf9011686-oauth-serving-cert\") pod \"console-6868d857cb-kcx5x\" (UID: \"e77492b2-9db5-4716-a2b4-78fcf9011686\") " pod="openshift-console/console-6868d857cb-kcx5x" Sep 10 13:19:59.683144 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:19:59.683121 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e77492b2-9db5-4716-a2b4-78fcf9011686-trusted-ca-bundle\") pod \"console-6868d857cb-kcx5x\" (UID: \"e77492b2-9db5-4716-a2b4-78fcf9011686\") " pod="openshift-console/console-6868d857cb-kcx5x" Sep 10 13:19:59.683184 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:19:59.683120 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/e77492b2-9db5-4716-a2b4-78fcf9011686-service-ca\") pod \"console-6868d857cb-kcx5x\" (UID: \"e77492b2-9db5-4716-a2b4-78fcf9011686\") " pod="openshift-console/console-6868d857cb-kcx5x" Sep 10 13:19:59.684574 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:19:59.684556 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/e77492b2-9db5-4716-a2b4-78fcf9011686-console-oauth-config\") pod \"console-6868d857cb-kcx5x\" (UID: \"e77492b2-9db5-4716-a2b4-78fcf9011686\") " pod="openshift-console/console-6868d857cb-kcx5x" Sep 10 13:19:59.684710 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:19:59.684690 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/e77492b2-9db5-4716-a2b4-78fcf9011686-console-serving-cert\") pod \"console-6868d857cb-kcx5x\" (UID: \"e77492b2-9db5-4716-a2b4-78fcf9011686\") " pod="openshift-console/console-6868d857cb-kcx5x" Sep 10 13:19:59.692990 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:19:59.692964 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vbpdx\" (UniqueName: \"kubernetes.io/projected/e77492b2-9db5-4716-a2b4-78fcf9011686-kube-api-access-vbpdx\") pod \"console-6868d857cb-kcx5x\" (UID: \"e77492b2-9db5-4716-a2b4-78fcf9011686\") " pod="openshift-console/console-6868d857cb-kcx5x" Sep 10 13:19:59.845482 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:19:59.845364 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6868d857cb-kcx5x" Sep 10 13:20:00.122736 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:20:00.122650 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-6868d857cb-kcx5x"] Sep 10 13:20:00.127354 ip-10-0-139-174 kubenswrapper[2557]: W0910 13:20:00.127324 2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode77492b2_9db5_4716_a2b4_78fcf9011686.slice/crio-caa8fb3e0252f7057b9d780a249beb22925809abd37e66e70fedc5146cdfe92c WatchSource:0}: Error finding container caa8fb3e0252f7057b9d780a249beb22925809abd37e66e70fedc5146cdfe92c: Status 404 returned error can't find the container with id caa8fb3e0252f7057b9d780a249beb22925809abd37e66e70fedc5146cdfe92c Sep 10 13:20:00.972961 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:20:00.972922 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6868d857cb-kcx5x" event={"ID":"e77492b2-9db5-4716-a2b4-78fcf9011686","Type":"ContainerStarted","Data":"ad002245a088c6235e42c21dc1fbe4c2707fa95b948569ae4c8a8a3f68d78c48"} Sep 10 13:20:00.973448 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:20:00.972986 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6868d857cb-kcx5x" event={"ID":"e77492b2-9db5-4716-a2b4-78fcf9011686","Type":"ContainerStarted","Data":"caa8fb3e0252f7057b9d780a249beb22925809abd37e66e70fedc5146cdfe92c"} Sep 10 13:20:01.002769 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:20:01.002715 2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-6868d857cb-kcx5x" podStartSLOduration=2.002698554 podStartE2EDuration="2.002698554s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 13:19:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 13:20:01.000838833 +0000 UTC m=+161.307833280" watchObservedRunningTime="2026-09-10 13:20:01.002698554 +0000 UTC m=+161.309692968" Sep 10 13:20:09.846033 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:20:09.845996 2557 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-6868d857cb-kcx5x" Sep 10 13:20:09.846392 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:20:09.846106 2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-6868d857cb-kcx5x" Sep 10 13:20:09.850672 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:20:09.850651 2557 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-6868d857cb-kcx5x" Sep 10 13:20:10.013360 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:20:10.013334 2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-6868d857cb-kcx5x" Sep 10 13:20:10.074963 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:20:10.074938 2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-6c9ff75b79-gzl7j"] Sep 10 13:20:35.099859 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:20:35.099819 2557 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-6c9ff75b79-gzl7j" podUID="ad02681a-5dfa-4abd-9ee4-6937c1c6e26a" containerName="console" containerID="cri-o://8f3e3372567439a923d097ab61190d67272e1791829ee7256bec21da74eabaed" gracePeriod=15 Sep 10 13:20:35.337700 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:20:35.337678 2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6c9ff75b79-gzl7j" Sep 10 13:20:35.435719 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:20:35.435656 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/ad02681a-5dfa-4abd-9ee4-6937c1c6e26a-console-config\") pod \"ad02681a-5dfa-4abd-9ee4-6937c1c6e26a\" (UID: \"ad02681a-5dfa-4abd-9ee4-6937c1c6e26a\") " Sep 10 13:20:35.435719 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:20:35.435695 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/ad02681a-5dfa-4abd-9ee4-6937c1c6e26a-oauth-serving-cert\") pod \"ad02681a-5dfa-4abd-9ee4-6937c1c6e26a\" (UID: \"ad02681a-5dfa-4abd-9ee4-6937c1c6e26a\") " Sep 10 13:20:35.435719 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:20:35.435714 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ad02681a-5dfa-4abd-9ee4-6937c1c6e26a-trusted-ca-bundle\") pod \"ad02681a-5dfa-4abd-9ee4-6937c1c6e26a\" (UID: \"ad02681a-5dfa-4abd-9ee4-6937c1c6e26a\") " Sep 10 13:20:35.435934 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:20:35.435748 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/ad02681a-5dfa-4abd-9ee4-6937c1c6e26a-console-oauth-config\") pod \"ad02681a-5dfa-4abd-9ee4-6937c1c6e26a\" (UID: \"ad02681a-5dfa-4abd-9ee4-6937c1c6e26a\") " Sep 10 13:20:35.435934 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:20:35.435782 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/ad02681a-5dfa-4abd-9ee4-6937c1c6e26a-console-serving-cert\") pod \"ad02681a-5dfa-4abd-9ee4-6937c1c6e26a\" (UID: \"ad02681a-5dfa-4abd-9ee4-6937c1c6e26a\") " Sep 10 13:20:35.435934 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:20:35.435816 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7tvk4\" (UniqueName: \"kubernetes.io/projected/ad02681a-5dfa-4abd-9ee4-6937c1c6e26a-kube-api-access-7tvk4\") pod \"ad02681a-5dfa-4abd-9ee4-6937c1c6e26a\" (UID: \"ad02681a-5dfa-4abd-9ee4-6937c1c6e26a\") " Sep 10 13:20:35.435934 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:20:35.435853 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/ad02681a-5dfa-4abd-9ee4-6937c1c6e26a-service-ca\") pod \"ad02681a-5dfa-4abd-9ee4-6937c1c6e26a\" (UID: \"ad02681a-5dfa-4abd-9ee4-6937c1c6e26a\") " Sep 10 13:20:35.436128 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:20:35.436067 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ad02681a-5dfa-4abd-9ee4-6937c1c6e26a-console-config" (OuterVolumeSpecName: "console-config") pod "ad02681a-5dfa-4abd-9ee4-6937c1c6e26a" (UID: "ad02681a-5dfa-4abd-9ee4-6937c1c6e26a"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 10 13:20:35.436179 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:20:35.436125 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ad02681a-5dfa-4abd-9ee4-6937c1c6e26a-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "ad02681a-5dfa-4abd-9ee4-6937c1c6e26a" (UID: "ad02681a-5dfa-4abd-9ee4-6937c1c6e26a"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 10 13:20:35.436179 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:20:35.436151 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ad02681a-5dfa-4abd-9ee4-6937c1c6e26a-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "ad02681a-5dfa-4abd-9ee4-6937c1c6e26a" (UID: "ad02681a-5dfa-4abd-9ee4-6937c1c6e26a"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 10 13:20:35.436352 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:20:35.436332 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ad02681a-5dfa-4abd-9ee4-6937c1c6e26a-service-ca" (OuterVolumeSpecName: "service-ca") pod "ad02681a-5dfa-4abd-9ee4-6937c1c6e26a" (UID: "ad02681a-5dfa-4abd-9ee4-6937c1c6e26a"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 10 13:20:35.438126 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:20:35.438098 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ad02681a-5dfa-4abd-9ee4-6937c1c6e26a-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "ad02681a-5dfa-4abd-9ee4-6937c1c6e26a" (UID: "ad02681a-5dfa-4abd-9ee4-6937c1c6e26a"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 10 13:20:35.438223 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:20:35.438130 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ad02681a-5dfa-4abd-9ee4-6937c1c6e26a-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "ad02681a-5dfa-4abd-9ee4-6937c1c6e26a" (UID: "ad02681a-5dfa-4abd-9ee4-6937c1c6e26a"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 10 13:20:35.438223 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:20:35.438135 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ad02681a-5dfa-4abd-9ee4-6937c1c6e26a-kube-api-access-7tvk4" (OuterVolumeSpecName: "kube-api-access-7tvk4") pod "ad02681a-5dfa-4abd-9ee4-6937c1c6e26a" (UID: "ad02681a-5dfa-4abd-9ee4-6937c1c6e26a"). InnerVolumeSpecName "kube-api-access-7tvk4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 10 13:20:35.536753 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:20:35.536722 2557 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/ad02681a-5dfa-4abd-9ee4-6937c1c6e26a-service-ca\") on node \"ip-10-0-139-174.ec2.internal\" DevicePath \"\"" Sep 10 13:20:35.536753 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:20:35.536751 2557 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/ad02681a-5dfa-4abd-9ee4-6937c1c6e26a-console-config\") on node \"ip-10-0-139-174.ec2.internal\" DevicePath \"\"" Sep 10 13:20:35.536753 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:20:35.536761 2557 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/ad02681a-5dfa-4abd-9ee4-6937c1c6e26a-oauth-serving-cert\") on node \"ip-10-0-139-174.ec2.internal\" DevicePath \"\"" Sep 10 13:20:35.536926 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:20:35.536770 2557 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ad02681a-5dfa-4abd-9ee4-6937c1c6e26a-trusted-ca-bundle\") on node \"ip-10-0-139-174.ec2.internal\" DevicePath \"\"" Sep 10 13:20:35.536926 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:20:35.536779 2557 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/ad02681a-5dfa-4abd-9ee4-6937c1c6e26a-console-oauth-config\") on node \"ip-10-0-139-174.ec2.internal\" DevicePath \"\"" Sep 10 13:20:35.536926 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:20:35.536787 2557 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/ad02681a-5dfa-4abd-9ee4-6937c1c6e26a-console-serving-cert\") on node \"ip-10-0-139-174.ec2.internal\" DevicePath \"\"" Sep 10 13:20:35.536926 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:20:35.536795 2557 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7tvk4\" (UniqueName: \"kubernetes.io/projected/ad02681a-5dfa-4abd-9ee4-6937c1c6e26a-kube-api-access-7tvk4\") on node \"ip-10-0-139-174.ec2.internal\" DevicePath \"\"" Sep 10 13:20:36.101731 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:20:36.101694 2557 generic.go:356] "Generic (PLEG): container finished" podID="ad02681a-5dfa-4abd-9ee4-6937c1c6e26a" containerID="8f3e3372567439a923d097ab61190d67272e1791829ee7256bec21da74eabaed" exitCode=0 Sep 10 13:20:36.102101 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:20:36.101756 2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6c9ff75b79-gzl7j" Sep 10 13:20:36.102101 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:20:36.101788 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6c9ff75b79-gzl7j" event={"ID":"ad02681a-5dfa-4abd-9ee4-6937c1c6e26a","Type":"ContainerDied","Data":"8f3e3372567439a923d097ab61190d67272e1791829ee7256bec21da74eabaed"} Sep 10 13:20:36.102101 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:20:36.101879 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6c9ff75b79-gzl7j" event={"ID":"ad02681a-5dfa-4abd-9ee4-6937c1c6e26a","Type":"ContainerDied","Data":"84b8ee716bd8883708be9b3e51822a449c8e7cc7026761bf6733cb29e6990dcd"} Sep 10 13:20:36.102101 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:20:36.101900 2557 scope.go:117] "RemoveContainer" containerID="8f3e3372567439a923d097ab61190d67272e1791829ee7256bec21da74eabaed" Sep 10 13:20:36.110311 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:20:36.110294 2557 scope.go:117] "RemoveContainer" containerID="8f3e3372567439a923d097ab61190d67272e1791829ee7256bec21da74eabaed" Sep 10 13:20:36.110607 ip-10-0-139-174 kubenswrapper[2557]: E0910 13:20:36.110589 2557 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8f3e3372567439a923d097ab61190d67272e1791829ee7256bec21da74eabaed\": container with ID starting with 8f3e3372567439a923d097ab61190d67272e1791829ee7256bec21da74eabaed not found: ID does not exist" containerID="8f3e3372567439a923d097ab61190d67272e1791829ee7256bec21da74eabaed" Sep 10 13:20:36.110660 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:20:36.110617 2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8f3e3372567439a923d097ab61190d67272e1791829ee7256bec21da74eabaed"} err="failed to get container status \"8f3e3372567439a923d097ab61190d67272e1791829ee7256bec21da74eabaed\": rpc error: code = NotFound desc = could not find container \"8f3e3372567439a923d097ab61190d67272e1791829ee7256bec21da74eabaed\": container with ID starting with 8f3e3372567439a923d097ab61190d67272e1791829ee7256bec21da74eabaed not found: ID does not exist" Sep 10 13:20:36.121786 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:20:36.121763 2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-6c9ff75b79-gzl7j"] Sep 10 13:20:36.125034 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:20:36.125013 2557 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-6c9ff75b79-gzl7j"] Sep 10 13:20:36.218298 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:20:36.218272 2557 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ad02681a-5dfa-4abd-9ee4-6937c1c6e26a" path="/var/lib/kubelet/pods/ad02681a-5dfa-4abd-9ee4-6937c1c6e26a/volumes" Sep 10 13:20:42.331956 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:20:42.331925 2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540bnv9z"] Sep 10 13:20:42.332381 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:20:42.332196 2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ad02681a-5dfa-4abd-9ee4-6937c1c6e26a" containerName="console" Sep 10 13:20:42.332381 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:20:42.332207 2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="ad02681a-5dfa-4abd-9ee4-6937c1c6e26a" containerName="console" Sep 10 13:20:42.332381 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:20:42.332249 2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="ad02681a-5dfa-4abd-9ee4-6937c1c6e26a" containerName="console" Sep 10 13:20:42.334377 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:20:42.334356 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540bnv9z" Sep 10 13:20:42.336671 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:20:42.336653 2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\"" Sep 10 13:20:42.337090 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:20:42.337078 2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\"" Sep 10 13:20:42.337149 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:20:42.337117 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-9r2bh\"" Sep 10 13:20:42.354633 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:20:42.354606 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540bnv9z"] Sep 10 13:20:42.485009 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:20:42.484979 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/83c61a8d-10aa-481b-8ace-48215a378ef1-util\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540bnv9z\" (UID: \"83c61a8d-10aa-481b-8ace-48215a378ef1\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540bnv9z" Sep 10 13:20:42.485125 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:20:42.485024 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/83c61a8d-10aa-481b-8ace-48215a378ef1-bundle\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540bnv9z\" (UID: \"83c61a8d-10aa-481b-8ace-48215a378ef1\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540bnv9z" Sep 10 13:20:42.485125 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:20:42.485053 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-79dz2\" (UniqueName: \"kubernetes.io/projected/83c61a8d-10aa-481b-8ace-48215a378ef1-kube-api-access-79dz2\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540bnv9z\" (UID: \"83c61a8d-10aa-481b-8ace-48215a378ef1\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540bnv9z" Sep 10 13:20:42.585541 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:20:42.585481 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/83c61a8d-10aa-481b-8ace-48215a378ef1-util\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540bnv9z\" (UID: \"83c61a8d-10aa-481b-8ace-48215a378ef1\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540bnv9z" Sep 10 13:20:42.585541 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:20:42.585523 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/83c61a8d-10aa-481b-8ace-48215a378ef1-bundle\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540bnv9z\" (UID: \"83c61a8d-10aa-481b-8ace-48215a378ef1\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540bnv9z" Sep 10 13:20:42.585661 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:20:42.585553 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-79dz2\" (UniqueName: \"kubernetes.io/projected/83c61a8d-10aa-481b-8ace-48215a378ef1-kube-api-access-79dz2\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540bnv9z\" (UID: \"83c61a8d-10aa-481b-8ace-48215a378ef1\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540bnv9z" Sep 10 13:20:42.585827 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:20:42.585808 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/83c61a8d-10aa-481b-8ace-48215a378ef1-util\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540bnv9z\" (UID: \"83c61a8d-10aa-481b-8ace-48215a378ef1\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540bnv9z" Sep 10 13:20:42.585893 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:20:42.585877 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/83c61a8d-10aa-481b-8ace-48215a378ef1-bundle\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540bnv9z\" (UID: \"83c61a8d-10aa-481b-8ace-48215a378ef1\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540bnv9z" Sep 10 13:20:42.594094 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:20:42.594063 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-79dz2\" (UniqueName: \"kubernetes.io/projected/83c61a8d-10aa-481b-8ace-48215a378ef1-kube-api-access-79dz2\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540bnv9z\" (UID: \"83c61a8d-10aa-481b-8ace-48215a378ef1\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540bnv9z" Sep 10 13:20:42.642874 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:20:42.642846 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540bnv9z" Sep 10 13:20:42.918160 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:20:42.918087 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540bnv9z"] Sep 10 13:20:42.921026 ip-10-0-139-174 kubenswrapper[2557]: W0910 13:20:42.920999 2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod83c61a8d_10aa_481b_8ace_48215a378ef1.slice/crio-4e5f07e73ff3e2b0f06f67433f83656a7137e005ede4d7dd8c53dacc242ab607 WatchSource:0}: Error finding container 4e5f07e73ff3e2b0f06f67433f83656a7137e005ede4d7dd8c53dacc242ab607: Status 404 returned error can't find the container with id 4e5f07e73ff3e2b0f06f67433f83656a7137e005ede4d7dd8c53dacc242ab607 Sep 10 13:20:43.133470 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:20:43.133438 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540bnv9z" event={"ID":"83c61a8d-10aa-481b-8ace-48215a378ef1","Type":"ContainerStarted","Data":"4e5f07e73ff3e2b0f06f67433f83656a7137e005ede4d7dd8c53dacc242ab607"} Sep 10 13:20:50.162447 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:20:50.162335 2557 generic.go:356] "Generic (PLEG): container finished" podID="83c61a8d-10aa-481b-8ace-48215a378ef1" containerID="48c93e1a8e2dd9f8140bb66f95a790b24ff29dfa61561f90f414ee6c2cc50bb8" exitCode=0 Sep 10 13:20:50.162786 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:20:50.162443 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540bnv9z" event={"ID":"83c61a8d-10aa-481b-8ace-48215a378ef1","Type":"ContainerDied","Data":"48c93e1a8e2dd9f8140bb66f95a790b24ff29dfa61561f90f414ee6c2cc50bb8"} Sep 10 13:20:52.172300 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:20:52.172272 2557 generic.go:356] "Generic (PLEG): container finished" podID="83c61a8d-10aa-481b-8ace-48215a378ef1" containerID="bd838597442576a6ca56758a2ba8d9b0cc5ee900c2dd8e8276723465434038a7" exitCode=0 Sep 10 13:20:52.172694 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:20:52.172360 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540bnv9z" event={"ID":"83c61a8d-10aa-481b-8ace-48215a378ef1","Type":"ContainerDied","Data":"bd838597442576a6ca56758a2ba8d9b0cc5ee900c2dd8e8276723465434038a7"} Sep 10 13:21:02.214342 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:21:02.214304 2557 generic.go:356] "Generic (PLEG): container finished" podID="83c61a8d-10aa-481b-8ace-48215a378ef1" containerID="3cd6b6bd3ba41d519c031a0326a696d04bd21995ecfb785f5c83666fb44d75e4" exitCode=0 Sep 10 13:21:02.217673 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:21:02.217649 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540bnv9z" event={"ID":"83c61a8d-10aa-481b-8ace-48215a378ef1","Type":"ContainerDied","Data":"3cd6b6bd3ba41d519c031a0326a696d04bd21995ecfb785f5c83666fb44d75e4"} Sep 10 13:21:03.339186 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:21:03.339161 2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540bnv9z" Sep 10 13:21:03.438690 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:21:03.438663 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/83c61a8d-10aa-481b-8ace-48215a378ef1-util\") pod \"83c61a8d-10aa-481b-8ace-48215a378ef1\" (UID: \"83c61a8d-10aa-481b-8ace-48215a378ef1\") " Sep 10 13:21:03.438796 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:21:03.438692 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/83c61a8d-10aa-481b-8ace-48215a378ef1-bundle\") pod \"83c61a8d-10aa-481b-8ace-48215a378ef1\" (UID: \"83c61a8d-10aa-481b-8ace-48215a378ef1\") " Sep 10 13:21:03.438796 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:21:03.438735 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-79dz2\" (UniqueName: \"kubernetes.io/projected/83c61a8d-10aa-481b-8ace-48215a378ef1-kube-api-access-79dz2\") pod \"83c61a8d-10aa-481b-8ace-48215a378ef1\" (UID: \"83c61a8d-10aa-481b-8ace-48215a378ef1\") " Sep 10 13:21:03.439313 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:21:03.439280 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/83c61a8d-10aa-481b-8ace-48215a378ef1-bundle" (OuterVolumeSpecName: "bundle") pod "83c61a8d-10aa-481b-8ace-48215a378ef1" (UID: "83c61a8d-10aa-481b-8ace-48215a378ef1"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 13:21:03.440942 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:21:03.440915 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/83c61a8d-10aa-481b-8ace-48215a378ef1-kube-api-access-79dz2" (OuterVolumeSpecName: "kube-api-access-79dz2") pod "83c61a8d-10aa-481b-8ace-48215a378ef1" (UID: "83c61a8d-10aa-481b-8ace-48215a378ef1"). InnerVolumeSpecName "kube-api-access-79dz2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 10 13:21:03.442998 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:21:03.442979 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/83c61a8d-10aa-481b-8ace-48215a378ef1-util" (OuterVolumeSpecName: "util") pod "83c61a8d-10aa-481b-8ace-48215a378ef1" (UID: "83c61a8d-10aa-481b-8ace-48215a378ef1"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 13:21:03.539154 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:21:03.539099 2557 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/83c61a8d-10aa-481b-8ace-48215a378ef1-util\") on node \"ip-10-0-139-174.ec2.internal\" DevicePath \"\"" Sep 10 13:21:03.539154 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:21:03.539146 2557 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/83c61a8d-10aa-481b-8ace-48215a378ef1-bundle\") on node \"ip-10-0-139-174.ec2.internal\" DevicePath \"\"" Sep 10 13:21:03.539284 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:21:03.539157 2557 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-79dz2\" (UniqueName: \"kubernetes.io/projected/83c61a8d-10aa-481b-8ace-48215a378ef1-kube-api-access-79dz2\") on node \"ip-10-0-139-174.ec2.internal\" DevicePath \"\"" Sep 10 13:21:04.222521 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:21:04.222482 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540bnv9z" event={"ID":"83c61a8d-10aa-481b-8ace-48215a378ef1","Type":"ContainerDied","Data":"4e5f07e73ff3e2b0f06f67433f83656a7137e005ede4d7dd8c53dacc242ab607"} Sep 10 13:21:04.222521 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:21:04.222529 2557 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="4e5f07e73ff3e2b0f06f67433f83656a7137e005ede4d7dd8c53dacc242ab607" Sep 10 13:21:04.222777 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:21:04.222566 2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540bnv9z" Sep 10 13:21:11.406112 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:21:11.406079 2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-vdprm"] Sep 10 13:21:11.406523 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:21:11.406335 2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="83c61a8d-10aa-481b-8ace-48215a378ef1" containerName="util" Sep 10 13:21:11.406523 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:21:11.406345 2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="83c61a8d-10aa-481b-8ace-48215a378ef1" containerName="util" Sep 10 13:21:11.406523 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:21:11.406366 2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="83c61a8d-10aa-481b-8ace-48215a378ef1" containerName="extract" Sep 10 13:21:11.406523 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:21:11.406371 2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="83c61a8d-10aa-481b-8ace-48215a378ef1" containerName="extract" Sep 10 13:21:11.406523 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:21:11.406378 2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="83c61a8d-10aa-481b-8ace-48215a378ef1" containerName="pull" Sep 10 13:21:11.406523 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:21:11.406383 2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="83c61a8d-10aa-481b-8ace-48215a378ef1" containerName="pull" Sep 10 13:21:11.406523 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:21:11.406437 2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="83c61a8d-10aa-481b-8ace-48215a378ef1" containerName="extract" Sep 10 13:21:11.408292 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:21:11.408269 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-vdprm" Sep 10 13:21:11.410619 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:21:11.410601 2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager-operator\"/\"kube-root-ca.crt\"" Sep 10 13:21:11.410715 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:21:11.410664 2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager-operator\"/\"openshift-service-ca.crt\"" Sep 10 13:21:11.410754 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:21:11.410723 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager-operator\"/\"cert-manager-operator-controller-manager-dockercfg-n262x\"" Sep 10 13:21:11.444838 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:21:11.444808 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-vdprm"] Sep 10 13:21:11.492510 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:21:11.492483 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9n7zh\" (UniqueName: \"kubernetes.io/projected/d961431a-5330-46b2-9d30-32e6a561e818-kube-api-access-9n7zh\") pod \"cert-manager-operator-controller-manager-55c444bf86-vdprm\" (UID: \"d961431a-5330-46b2-9d30-32e6a561e818\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-vdprm" Sep 10 13:21:11.492608 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:21:11.492526 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/d961431a-5330-46b2-9d30-32e6a561e818-tmp\") pod \"cert-manager-operator-controller-manager-55c444bf86-vdprm\" (UID: \"d961431a-5330-46b2-9d30-32e6a561e818\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-vdprm" Sep 10 13:21:11.593179 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:21:11.593152 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/d961431a-5330-46b2-9d30-32e6a561e818-tmp\") pod \"cert-manager-operator-controller-manager-55c444bf86-vdprm\" (UID: \"d961431a-5330-46b2-9d30-32e6a561e818\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-vdprm" Sep 10 13:21:11.593282 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:21:11.593232 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9n7zh\" (UniqueName: \"kubernetes.io/projected/d961431a-5330-46b2-9d30-32e6a561e818-kube-api-access-9n7zh\") pod \"cert-manager-operator-controller-manager-55c444bf86-vdprm\" (UID: \"d961431a-5330-46b2-9d30-32e6a561e818\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-vdprm" Sep 10 13:21:11.593600 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:21:11.593578 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/d961431a-5330-46b2-9d30-32e6a561e818-tmp\") pod \"cert-manager-operator-controller-manager-55c444bf86-vdprm\" (UID: \"d961431a-5330-46b2-9d30-32e6a561e818\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-vdprm" Sep 10 13:21:11.602379 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:21:11.602361 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9n7zh\" (UniqueName: \"kubernetes.io/projected/d961431a-5330-46b2-9d30-32e6a561e818-kube-api-access-9n7zh\") pod \"cert-manager-operator-controller-manager-55c444bf86-vdprm\" (UID: \"d961431a-5330-46b2-9d30-32e6a561e818\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-vdprm" Sep 10 13:21:11.716939 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:21:11.716916 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-vdprm" Sep 10 13:21:12.003882 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:21:12.003849 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-vdprm"] Sep 10 13:21:12.006700 ip-10-0-139-174 kubenswrapper[2557]: W0910 13:21:12.006664 2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd961431a_5330_46b2_9d30_32e6a561e818.slice/crio-355024630ae516c02b71a77dab5289710d4404391897687a4a7c0f3cd1a878bb WatchSource:0}: Error finding container 355024630ae516c02b71a77dab5289710d4404391897687a4a7c0f3cd1a878bb: Status 404 returned error can't find the container with id 355024630ae516c02b71a77dab5289710d4404391897687a4a7c0f3cd1a878bb Sep 10 13:21:12.251646 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:21:12.251613 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-vdprm" event={"ID":"d961431a-5330-46b2-9d30-32e6a561e818","Type":"ContainerStarted","Data":"355024630ae516c02b71a77dab5289710d4404391897687a4a7c0f3cd1a878bb"} Sep 10 13:21:14.262160 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:21:14.262125 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-vdprm" event={"ID":"d961431a-5330-46b2-9d30-32e6a561e818","Type":"ContainerStarted","Data":"0e90067d1be46e9ce91b56ec10eff5796fac4ede39c26ba58b8ffee401688c9d"} Sep 10 13:21:14.308800 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:21:14.308755 2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-vdprm" podStartSLOduration=1.783113523 podStartE2EDuration="3.308740184s" totalImagesPullingTime="1.525626661s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 13:21:11 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 13:21:14.306296006 +0000 UTC m=+234.613290421" watchObservedRunningTime="2026-09-10 13:21:14.308740184 +0000 UTC m=+234.615734672" Sep 10 13:21:14.353822 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:21:14.351822 2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-webhook-6486ffd997-7dlrw"] Sep 10 13:21:14.355094 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:21:14.355066 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-webhook-6486ffd997-7dlrw" Sep 10 13:21:14.357718 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:21:14.357684 2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"openshift-service-ca.crt\"" Sep 10 13:21:14.357880 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:21:14.357795 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-webhook-dockercfg-xw4bj\"" Sep 10 13:21:14.357986 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:21:14.357973 2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"kube-root-ca.crt\"" Sep 10 13:21:14.363731 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:21:14.363710 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-webhook-6486ffd997-7dlrw"] Sep 10 13:21:14.413169 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:21:14.413139 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sqd2k\" (UniqueName: \"kubernetes.io/projected/119ccfaf-c54d-4cc0-b1df-6e1cd4d4434d-kube-api-access-sqd2k\") pod \"cert-manager-webhook-6486ffd997-7dlrw\" (UID: \"119ccfaf-c54d-4cc0-b1df-6e1cd4d4434d\") " pod="cert-manager/cert-manager-webhook-6486ffd997-7dlrw" Sep 10 13:21:14.413279 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:21:14.413179 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/119ccfaf-c54d-4cc0-b1df-6e1cd4d4434d-bound-sa-token\") pod \"cert-manager-webhook-6486ffd997-7dlrw\" (UID: \"119ccfaf-c54d-4cc0-b1df-6e1cd4d4434d\") " pod="cert-manager/cert-manager-webhook-6486ffd997-7dlrw" Sep 10 13:21:14.514076 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:21:14.514021 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sqd2k\" (UniqueName: \"kubernetes.io/projected/119ccfaf-c54d-4cc0-b1df-6e1cd4d4434d-kube-api-access-sqd2k\") pod \"cert-manager-webhook-6486ffd997-7dlrw\" (UID: \"119ccfaf-c54d-4cc0-b1df-6e1cd4d4434d\") " pod="cert-manager/cert-manager-webhook-6486ffd997-7dlrw" Sep 10 13:21:14.514076 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:21:14.514057 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/119ccfaf-c54d-4cc0-b1df-6e1cd4d4434d-bound-sa-token\") pod \"cert-manager-webhook-6486ffd997-7dlrw\" (UID: \"119ccfaf-c54d-4cc0-b1df-6e1cd4d4434d\") " pod="cert-manager/cert-manager-webhook-6486ffd997-7dlrw" Sep 10 13:21:14.525589 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:21:14.525566 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/119ccfaf-c54d-4cc0-b1df-6e1cd4d4434d-bound-sa-token\") pod \"cert-manager-webhook-6486ffd997-7dlrw\" (UID: \"119ccfaf-c54d-4cc0-b1df-6e1cd4d4434d\") " pod="cert-manager/cert-manager-webhook-6486ffd997-7dlrw" Sep 10 13:21:14.525684 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:21:14.525639 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sqd2k\" (UniqueName: \"kubernetes.io/projected/119ccfaf-c54d-4cc0-b1df-6e1cd4d4434d-kube-api-access-sqd2k\") pod \"cert-manager-webhook-6486ffd997-7dlrw\" (UID: \"119ccfaf-c54d-4cc0-b1df-6e1cd4d4434d\") " pod="cert-manager/cert-manager-webhook-6486ffd997-7dlrw" Sep 10 13:21:14.673864 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:21:14.673837 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-webhook-6486ffd997-7dlrw" Sep 10 13:21:14.968626 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:21:14.968438 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-webhook-6486ffd997-7dlrw"] Sep 10 13:21:14.971231 ip-10-0-139-174 kubenswrapper[2557]: W0910 13:21:14.971202 2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod119ccfaf_c54d_4cc0_b1df_6e1cd4d4434d.slice/crio-877473fb46dd68c46bbebb6a8deb622fcde9bee85b08a98c1f3fe5d9168002d7 WatchSource:0}: Error finding container 877473fb46dd68c46bbebb6a8deb622fcde9bee85b08a98c1f3fe5d9168002d7: Status 404 returned error can't find the container with id 877473fb46dd68c46bbebb6a8deb622fcde9bee85b08a98c1f3fe5d9168002d7 Sep 10 13:21:15.077227 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:21:15.077197 2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fjphcf"] Sep 10 13:21:15.080498 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:21:15.080481 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fjphcf" Sep 10 13:21:15.082588 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:21:15.082567 2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\"" Sep 10 13:21:15.082669 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:21:15.082589 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-9r2bh\"" Sep 10 13:21:15.082911 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:21:15.082891 2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\"" Sep 10 13:21:15.091190 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:21:15.091170 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fjphcf"] Sep 10 13:21:15.118953 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:21:15.118933 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/6a7e424b-5949-4c32-97fd-10d0d25ff4b8-util\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fjphcf\" (UID: \"6a7e424b-5949-4c32-97fd-10d0d25ff4b8\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fjphcf" Sep 10 13:21:15.119058 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:21:15.118972 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/6a7e424b-5949-4c32-97fd-10d0d25ff4b8-bundle\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fjphcf\" (UID: \"6a7e424b-5949-4c32-97fd-10d0d25ff4b8\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fjphcf" Sep 10 13:21:15.119058 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:21:15.118995 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ftgf2\" (UniqueName: \"kubernetes.io/projected/6a7e424b-5949-4c32-97fd-10d0d25ff4b8-kube-api-access-ftgf2\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fjphcf\" (UID: \"6a7e424b-5949-4c32-97fd-10d0d25ff4b8\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fjphcf" Sep 10 13:21:15.220276 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:21:15.220214 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ftgf2\" (UniqueName: \"kubernetes.io/projected/6a7e424b-5949-4c32-97fd-10d0d25ff4b8-kube-api-access-ftgf2\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fjphcf\" (UID: \"6a7e424b-5949-4c32-97fd-10d0d25ff4b8\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fjphcf" Sep 10 13:21:15.220276 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:21:15.220265 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/6a7e424b-5949-4c32-97fd-10d0d25ff4b8-util\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fjphcf\" (UID: \"6a7e424b-5949-4c32-97fd-10d0d25ff4b8\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fjphcf" Sep 10 13:21:15.220486 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:21:15.220465 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/6a7e424b-5949-4c32-97fd-10d0d25ff4b8-bundle\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fjphcf\" (UID: \"6a7e424b-5949-4c32-97fd-10d0d25ff4b8\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fjphcf" Sep 10 13:21:15.220617 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:21:15.220603 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/6a7e424b-5949-4c32-97fd-10d0d25ff4b8-util\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fjphcf\" (UID: \"6a7e424b-5949-4c32-97fd-10d0d25ff4b8\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fjphcf" Sep 10 13:21:15.220766 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:21:15.220750 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/6a7e424b-5949-4c32-97fd-10d0d25ff4b8-bundle\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fjphcf\" (UID: \"6a7e424b-5949-4c32-97fd-10d0d25ff4b8\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fjphcf" Sep 10 13:21:15.229158 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:21:15.229127 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ftgf2\" (UniqueName: \"kubernetes.io/projected/6a7e424b-5949-4c32-97fd-10d0d25ff4b8-kube-api-access-ftgf2\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fjphcf\" (UID: \"6a7e424b-5949-4c32-97fd-10d0d25ff4b8\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fjphcf" Sep 10 13:21:15.268020 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:21:15.267994 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-webhook-6486ffd997-7dlrw" event={"ID":"119ccfaf-c54d-4cc0-b1df-6e1cd4d4434d","Type":"ContainerStarted","Data":"877473fb46dd68c46bbebb6a8deb622fcde9bee85b08a98c1f3fe5d9168002d7"} Sep 10 13:21:15.389241 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:21:15.389217 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fjphcf" Sep 10 13:21:15.671731 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:21:15.671639 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fjphcf"] Sep 10 13:21:15.675816 ip-10-0-139-174 kubenswrapper[2557]: W0910 13:21:15.675786 2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6a7e424b_5949_4c32_97fd_10d0d25ff4b8.slice/crio-b6616f5c36deac8023813e4718e297145e51fc2f7d3235415dc468810d335373 WatchSource:0}: Error finding container b6616f5c36deac8023813e4718e297145e51fc2f7d3235415dc468810d335373: Status 404 returned error can't find the container with id b6616f5c36deac8023813e4718e297145e51fc2f7d3235415dc468810d335373 Sep 10 13:21:16.277076 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:21:16.277041 2557 generic.go:356] "Generic (PLEG): container finished" podID="6a7e424b-5949-4c32-97fd-10d0d25ff4b8" containerID="a5bff4dcda153ed7968fdc1fa2d0c8b8c3c156619791b91971e97efefb4a4113" exitCode=0 Sep 10 13:21:16.277532 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:21:16.277075 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fjphcf" event={"ID":"6a7e424b-5949-4c32-97fd-10d0d25ff4b8","Type":"ContainerDied","Data":"a5bff4dcda153ed7968fdc1fa2d0c8b8c3c156619791b91971e97efefb4a4113"} Sep 10 13:21:16.277532 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:21:16.277132 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fjphcf" event={"ID":"6a7e424b-5949-4c32-97fd-10d0d25ff4b8","Type":"ContainerStarted","Data":"b6616f5c36deac8023813e4718e297145e51fc2f7d3235415dc468810d335373"} Sep 10 13:21:17.437536 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:21:17.437503 2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-cainjector-559cdd8c87-k2hgx"] Sep 10 13:21:17.439525 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:21:17.439504 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-cainjector-559cdd8c87-k2hgx" Sep 10 13:21:17.441505 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:21:17.441484 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-cainjector-dockercfg-zmjnk\"" Sep 10 13:21:17.453045 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:21:17.453026 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-cainjector-559cdd8c87-k2hgx"] Sep 10 13:21:17.541289 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:21:17.541263 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/3a269308-b242-400d-a4f1-3bff90022477-bound-sa-token\") pod \"cert-manager-cainjector-559cdd8c87-k2hgx\" (UID: \"3a269308-b242-400d-a4f1-3bff90022477\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-k2hgx" Sep 10 13:21:17.541394 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:21:17.541295 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5vwgk\" (UniqueName: \"kubernetes.io/projected/3a269308-b242-400d-a4f1-3bff90022477-kube-api-access-5vwgk\") pod \"cert-manager-cainjector-559cdd8c87-k2hgx\" (UID: \"3a269308-b242-400d-a4f1-3bff90022477\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-k2hgx" Sep 10 13:21:17.642107 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:21:17.642045 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/3a269308-b242-400d-a4f1-3bff90022477-bound-sa-token\") pod \"cert-manager-cainjector-559cdd8c87-k2hgx\" (UID: \"3a269308-b242-400d-a4f1-3bff90022477\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-k2hgx" Sep 10 13:21:17.642107 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:21:17.642077 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5vwgk\" (UniqueName: \"kubernetes.io/projected/3a269308-b242-400d-a4f1-3bff90022477-kube-api-access-5vwgk\") pod \"cert-manager-cainjector-559cdd8c87-k2hgx\" (UID: \"3a269308-b242-400d-a4f1-3bff90022477\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-k2hgx" Sep 10 13:21:17.662327 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:21:17.662301 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/3a269308-b242-400d-a4f1-3bff90022477-bound-sa-token\") pod \"cert-manager-cainjector-559cdd8c87-k2hgx\" (UID: \"3a269308-b242-400d-a4f1-3bff90022477\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-k2hgx" Sep 10 13:21:17.663915 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:21:17.663895 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5vwgk\" (UniqueName: \"kubernetes.io/projected/3a269308-b242-400d-a4f1-3bff90022477-kube-api-access-5vwgk\") pod \"cert-manager-cainjector-559cdd8c87-k2hgx\" (UID: \"3a269308-b242-400d-a4f1-3bff90022477\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-k2hgx" Sep 10 13:21:17.748123 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:21:17.748099 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-cainjector-559cdd8c87-k2hgx" Sep 10 13:21:18.028818 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:21:18.028788 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-cainjector-559cdd8c87-k2hgx"] Sep 10 13:21:18.032277 ip-10-0-139-174 kubenswrapper[2557]: W0910 13:21:18.032241 2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod3a269308_b242_400d_a4f1_3bff90022477.slice/crio-716e3bd198db47f8498f0c51a8e9c1779572e4f5bfcb8b169c0a7466ffe3db20 WatchSource:0}: Error finding container 716e3bd198db47f8498f0c51a8e9c1779572e4f5bfcb8b169c0a7466ffe3db20: Status 404 returned error can't find the container with id 716e3bd198db47f8498f0c51a8e9c1779572e4f5bfcb8b169c0a7466ffe3db20 Sep 10 13:21:18.293168 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:21:18.293064 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-cainjector-559cdd8c87-k2hgx" event={"ID":"3a269308-b242-400d-a4f1-3bff90022477","Type":"ContainerStarted","Data":"a184d678f4ce31c0d9f9a01101e3dd3f87c257ed29936eb0d7e5712640f4ac47"} Sep 10 13:21:18.293168 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:21:18.293118 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-cainjector-559cdd8c87-k2hgx" event={"ID":"3a269308-b242-400d-a4f1-3bff90022477","Type":"ContainerStarted","Data":"716e3bd198db47f8498f0c51a8e9c1779572e4f5bfcb8b169c0a7466ffe3db20"} Sep 10 13:21:18.294934 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:21:18.294905 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-webhook-6486ffd997-7dlrw" event={"ID":"119ccfaf-c54d-4cc0-b1df-6e1cd4d4434d","Type":"ContainerStarted","Data":"08d6339b1e5df4d4627204b23648600c7aae9c350906a78c3983a5f0b5e36a32"} Sep 10 13:21:18.295056 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:21:18.295039 2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="cert-manager/cert-manager-webhook-6486ffd997-7dlrw" Sep 10 13:21:18.311327 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:21:18.311285 2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager/cert-manager-cainjector-559cdd8c87-k2hgx" podStartSLOduration=1.3112722350000001 podStartE2EDuration="1.311272235s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 13:21:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 13:21:18.309998986 +0000 UTC m=+238.616993413" watchObservedRunningTime="2026-09-10 13:21:18.311272235 +0000 UTC m=+238.618266648" Sep 10 13:21:18.331261 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:21:18.331209 2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager/cert-manager-webhook-6486ffd997-7dlrw" podStartSLOduration=1.963919297 podStartE2EDuration="4.331195654s" totalImagesPullingTime="2.367276357s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 13:21:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 13:21:18.329406878 +0000 UTC m=+238.636401304" watchObservedRunningTime="2026-09-10 13:21:18.331195654 +0000 UTC m=+238.638190067" Sep 10 13:21:24.301576 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:21:24.301547 2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="cert-manager/cert-manager-webhook-6486ffd997-7dlrw" Sep 10 13:21:27.389708 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:21:27.389676 2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-55b4d954d5-cww4z"] Sep 10 13:21:27.394867 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:21:27.394852 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-55b4d954d5-cww4z" Sep 10 13:21:27.397436 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:21:27.397384 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-dockercfg-wsv75\"" Sep 10 13:21:27.402714 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:21:27.402690 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-55b4d954d5-cww4z"] Sep 10 13:21:27.514662 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:21:27.514635 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/594a6108-498a-439e-b26c-a5cbadd89f6d-bound-sa-token\") pod \"cert-manager-55b4d954d5-cww4z\" (UID: \"594a6108-498a-439e-b26c-a5cbadd89f6d\") " pod="cert-manager/cert-manager-55b4d954d5-cww4z" Sep 10 13:21:27.514786 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:21:27.514673 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gt2s5\" (UniqueName: \"kubernetes.io/projected/594a6108-498a-439e-b26c-a5cbadd89f6d-kube-api-access-gt2s5\") pod \"cert-manager-55b4d954d5-cww4z\" (UID: \"594a6108-498a-439e-b26c-a5cbadd89f6d\") " pod="cert-manager/cert-manager-55b4d954d5-cww4z" Sep 10 13:21:27.616030 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:21:27.616003 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/594a6108-498a-439e-b26c-a5cbadd89f6d-bound-sa-token\") pod \"cert-manager-55b4d954d5-cww4z\" (UID: \"594a6108-498a-439e-b26c-a5cbadd89f6d\") " pod="cert-manager/cert-manager-55b4d954d5-cww4z" Sep 10 13:21:27.616148 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:21:27.616040 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gt2s5\" (UniqueName: \"kubernetes.io/projected/594a6108-498a-439e-b26c-a5cbadd89f6d-kube-api-access-gt2s5\") pod \"cert-manager-55b4d954d5-cww4z\" (UID: \"594a6108-498a-439e-b26c-a5cbadd89f6d\") " pod="cert-manager/cert-manager-55b4d954d5-cww4z" Sep 10 13:21:27.624850 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:21:27.624829 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/594a6108-498a-439e-b26c-a5cbadd89f6d-bound-sa-token\") pod \"cert-manager-55b4d954d5-cww4z\" (UID: \"594a6108-498a-439e-b26c-a5cbadd89f6d\") " pod="cert-manager/cert-manager-55b4d954d5-cww4z" Sep 10 13:21:27.624949 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:21:27.624916 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gt2s5\" (UniqueName: \"kubernetes.io/projected/594a6108-498a-439e-b26c-a5cbadd89f6d-kube-api-access-gt2s5\") pod \"cert-manager-55b4d954d5-cww4z\" (UID: \"594a6108-498a-439e-b26c-a5cbadd89f6d\") " pod="cert-manager/cert-manager-55b4d954d5-cww4z" Sep 10 13:21:27.704876 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:21:27.704856 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-55b4d954d5-cww4z" Sep 10 13:21:27.978612 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:21:27.978577 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-55b4d954d5-cww4z"] Sep 10 13:21:27.980807 ip-10-0-139-174 kubenswrapper[2557]: W0910 13:21:27.980780 2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod594a6108_498a_439e_b26c_a5cbadd89f6d.slice/crio-9589a161b6b08650930b87fd6c72476fe932d614a58ce1231125b15e86cd22e3 WatchSource:0}: Error finding container 9589a161b6b08650930b87fd6c72476fe932d614a58ce1231125b15e86cd22e3: Status 404 returned error can't find the container with id 9589a161b6b08650930b87fd6c72476fe932d614a58ce1231125b15e86cd22e3 Sep 10 13:21:28.334820 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:21:28.334732 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-55b4d954d5-cww4z" event={"ID":"594a6108-498a-439e-b26c-a5cbadd89f6d","Type":"ContainerStarted","Data":"e8327fa6fe1798275ba68318fddf08bcb57aa87cc0d2de1df86890001b587627"} Sep 10 13:21:28.334820 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:21:28.334769 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-55b4d954d5-cww4z" event={"ID":"594a6108-498a-439e-b26c-a5cbadd89f6d","Type":"ContainerStarted","Data":"9589a161b6b08650930b87fd6c72476fe932d614a58ce1231125b15e86cd22e3"} Sep 10 13:21:28.360689 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:21:28.360640 2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager/cert-manager-55b4d954d5-cww4z" podStartSLOduration=1.360625359 podStartE2EDuration="1.360625359s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 13:21:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 13:21:28.358283462 +0000 UTC m=+248.665277886" watchObservedRunningTime="2026-09-10 13:21:28.360625359 +0000 UTC m=+248.667619773" Sep 10 13:21:31.350288 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:21:31.350210 2557 generic.go:356] "Generic (PLEG): container finished" podID="6a7e424b-5949-4c32-97fd-10d0d25ff4b8" containerID="21cb40cfe11f1f33a5b5b2395c53856d8ae796be2685c8c5a980b395723c11d6" exitCode=0 Sep 10 13:21:31.350288 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:21:31.350268 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fjphcf" event={"ID":"6a7e424b-5949-4c32-97fd-10d0d25ff4b8","Type":"ContainerDied","Data":"21cb40cfe11f1f33a5b5b2395c53856d8ae796be2685c8c5a980b395723c11d6"} Sep 10 13:21:32.357073 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:21:32.357038 2557 generic.go:356] "Generic (PLEG): container finished" podID="6a7e424b-5949-4c32-97fd-10d0d25ff4b8" containerID="dfd0e385cacb40b078cc3a81ad5b1d862b2d5c8737445e559cabd710208dad27" exitCode=0 Sep 10 13:21:32.357503 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:21:32.357095 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fjphcf" event={"ID":"6a7e424b-5949-4c32-97fd-10d0d25ff4b8","Type":"ContainerDied","Data":"dfd0e385cacb40b078cc3a81ad5b1d862b2d5c8737445e559cabd710208dad27"} Sep 10 13:21:33.504246 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:21:33.504220 2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fjphcf" Sep 10 13:21:33.562397 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:21:33.562367 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/6a7e424b-5949-4c32-97fd-10d0d25ff4b8-util\") pod \"6a7e424b-5949-4c32-97fd-10d0d25ff4b8\" (UID: \"6a7e424b-5949-4c32-97fd-10d0d25ff4b8\") " Sep 10 13:21:33.562549 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:21:33.562409 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ftgf2\" (UniqueName: \"kubernetes.io/projected/6a7e424b-5949-4c32-97fd-10d0d25ff4b8-kube-api-access-ftgf2\") pod \"6a7e424b-5949-4c32-97fd-10d0d25ff4b8\" (UID: \"6a7e424b-5949-4c32-97fd-10d0d25ff4b8\") " Sep 10 13:21:33.562549 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:21:33.562464 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/6a7e424b-5949-4c32-97fd-10d0d25ff4b8-bundle\") pod \"6a7e424b-5949-4c32-97fd-10d0d25ff4b8\" (UID: \"6a7e424b-5949-4c32-97fd-10d0d25ff4b8\") " Sep 10 13:21:33.562880 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:21:33.562855 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6a7e424b-5949-4c32-97fd-10d0d25ff4b8-bundle" (OuterVolumeSpecName: "bundle") pod "6a7e424b-5949-4c32-97fd-10d0d25ff4b8" (UID: "6a7e424b-5949-4c32-97fd-10d0d25ff4b8"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 13:21:33.564636 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:21:33.564611 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6a7e424b-5949-4c32-97fd-10d0d25ff4b8-kube-api-access-ftgf2" (OuterVolumeSpecName: "kube-api-access-ftgf2") pod "6a7e424b-5949-4c32-97fd-10d0d25ff4b8" (UID: "6a7e424b-5949-4c32-97fd-10d0d25ff4b8"). InnerVolumeSpecName "kube-api-access-ftgf2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 10 13:21:33.566824 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:21:33.566803 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6a7e424b-5949-4c32-97fd-10d0d25ff4b8-util" (OuterVolumeSpecName: "util") pod "6a7e424b-5949-4c32-97fd-10d0d25ff4b8" (UID: "6a7e424b-5949-4c32-97fd-10d0d25ff4b8"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 13:21:33.663852 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:21:33.663799 2557 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/6a7e424b-5949-4c32-97fd-10d0d25ff4b8-util\") on node \"ip-10-0-139-174.ec2.internal\" DevicePath \"\"" Sep 10 13:21:33.663852 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:21:33.663821 2557 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ftgf2\" (UniqueName: \"kubernetes.io/projected/6a7e424b-5949-4c32-97fd-10d0d25ff4b8-kube-api-access-ftgf2\") on node \"ip-10-0-139-174.ec2.internal\" DevicePath \"\"" Sep 10 13:21:33.663852 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:21:33.663832 2557 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/6a7e424b-5949-4c32-97fd-10d0d25ff4b8-bundle\") on node \"ip-10-0-139-174.ec2.internal\" DevicePath \"\"" Sep 10 13:21:34.366449 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:21:34.366407 2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fjphcf" Sep 10 13:21:34.366586 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:21:34.366394 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fjphcf" event={"ID":"6a7e424b-5949-4c32-97fd-10d0d25ff4b8","Type":"ContainerDied","Data":"b6616f5c36deac8023813e4718e297145e51fc2f7d3235415dc468810d335373"} Sep 10 13:21:34.366586 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:21:34.366577 2557 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b6616f5c36deac8023813e4718e297145e51fc2f7d3235415dc468810d335373" Sep 10 13:21:44.801106 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:21:44.801030 2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af66txf"] Sep 10 13:21:44.801548 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:21:44.801320 2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6a7e424b-5949-4c32-97fd-10d0d25ff4b8" containerName="util" Sep 10 13:21:44.801548 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:21:44.801336 2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="6a7e424b-5949-4c32-97fd-10d0d25ff4b8" containerName="util" Sep 10 13:21:44.801548 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:21:44.801349 2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6a7e424b-5949-4c32-97fd-10d0d25ff4b8" containerName="extract" Sep 10 13:21:44.801548 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:21:44.801355 2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="6a7e424b-5949-4c32-97fd-10d0d25ff4b8" containerName="extract" Sep 10 13:21:44.801548 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:21:44.801365 2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6a7e424b-5949-4c32-97fd-10d0d25ff4b8" containerName="pull" Sep 10 13:21:44.801548 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:21:44.801372 2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="6a7e424b-5949-4c32-97fd-10d0d25ff4b8" containerName="pull" Sep 10 13:21:44.801548 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:21:44.801454 2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="6a7e424b-5949-4c32-97fd-10d0d25ff4b8" containerName="extract" Sep 10 13:21:44.803603 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:21:44.803585 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af66txf" Sep 10 13:21:44.809598 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:21:44.809568 2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\"" Sep 10 13:21:44.809727 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:21:44.809685 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-9r2bh\"" Sep 10 13:21:44.810325 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:21:44.810308 2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\"" Sep 10 13:21:44.876008 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:21:44.875977 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af66txf"] Sep 10 13:21:44.948650 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:21:44.948618 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/5cb1cf6e-21ba-4c37-9a0b-5a076bceaf9d-bundle\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af66txf\" (UID: \"5cb1cf6e-21ba-4c37-9a0b-5a076bceaf9d\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af66txf" Sep 10 13:21:44.948786 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:21:44.948690 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tc69b\" (UniqueName: \"kubernetes.io/projected/5cb1cf6e-21ba-4c37-9a0b-5a076bceaf9d-kube-api-access-tc69b\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af66txf\" (UID: \"5cb1cf6e-21ba-4c37-9a0b-5a076bceaf9d\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af66txf" Sep 10 13:21:44.948786 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:21:44.948732 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/5cb1cf6e-21ba-4c37-9a0b-5a076bceaf9d-util\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af66txf\" (UID: \"5cb1cf6e-21ba-4c37-9a0b-5a076bceaf9d\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af66txf" Sep 10 13:21:45.049390 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:21:45.049363 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/5cb1cf6e-21ba-4c37-9a0b-5a076bceaf9d-util\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af66txf\" (UID: \"5cb1cf6e-21ba-4c37-9a0b-5a076bceaf9d\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af66txf" Sep 10 13:21:45.049542 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:21:45.049408 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/5cb1cf6e-21ba-4c37-9a0b-5a076bceaf9d-bundle\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af66txf\" (UID: \"5cb1cf6e-21ba-4c37-9a0b-5a076bceaf9d\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af66txf" Sep 10 13:21:45.049542 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:21:45.049477 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tc69b\" (UniqueName: \"kubernetes.io/projected/5cb1cf6e-21ba-4c37-9a0b-5a076bceaf9d-kube-api-access-tc69b\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af66txf\" (UID: \"5cb1cf6e-21ba-4c37-9a0b-5a076bceaf9d\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af66txf" Sep 10 13:21:45.049778 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:21:45.049758 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/5cb1cf6e-21ba-4c37-9a0b-5a076bceaf9d-util\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af66txf\" (UID: \"5cb1cf6e-21ba-4c37-9a0b-5a076bceaf9d\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af66txf" Sep 10 13:21:45.049838 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:21:45.049818 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/5cb1cf6e-21ba-4c37-9a0b-5a076bceaf9d-bundle\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af66txf\" (UID: \"5cb1cf6e-21ba-4c37-9a0b-5a076bceaf9d\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af66txf" Sep 10 13:21:45.067441 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:21:45.067360 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tc69b\" (UniqueName: \"kubernetes.io/projected/5cb1cf6e-21ba-4c37-9a0b-5a076bceaf9d-kube-api-access-tc69b\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af66txf\" (UID: \"5cb1cf6e-21ba-4c37-9a0b-5a076bceaf9d\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af66txf" Sep 10 13:21:45.112319 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:21:45.112283 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af66txf" Sep 10 13:21:45.401573 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:21:45.401464 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af66txf"] Sep 10 13:21:45.404296 ip-10-0-139-174 kubenswrapper[2557]: W0910 13:21:45.404273 2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5cb1cf6e_21ba_4c37_9a0b_5a076bceaf9d.slice/crio-a769115a685e7cc779ea4c707a86a88424b85248d19a24edfbe9e98070875da9 WatchSource:0}: Error finding container a769115a685e7cc779ea4c707a86a88424b85248d19a24edfbe9e98070875da9: Status 404 returned error can't find the container with id a769115a685e7cc779ea4c707a86a88424b85248d19a24edfbe9e98070875da9 Sep 10 13:21:45.410938 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:21:45.410917 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af66txf" event={"ID":"5cb1cf6e-21ba-4c37-9a0b-5a076bceaf9d","Type":"ContainerStarted","Data":"a769115a685e7cc779ea4c707a86a88424b85248d19a24edfbe9e98070875da9"} Sep 10 13:21:46.416843 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:21:46.416751 2557 generic.go:356] "Generic (PLEG): container finished" podID="5cb1cf6e-21ba-4c37-9a0b-5a076bceaf9d" containerID="266ccbaadd1b86e51842224b6257eed13b22a53c256c361d224c0ad5b41aaa62" exitCode=0 Sep 10 13:21:46.416843 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:21:46.416821 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af66txf" event={"ID":"5cb1cf6e-21ba-4c37-9a0b-5a076bceaf9d","Type":"ContainerDied","Data":"266ccbaadd1b86e51842224b6257eed13b22a53c256c361d224c0ad5b41aaa62"} Sep 10 13:21:47.422643 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:21:47.422567 2557 generic.go:356] "Generic (PLEG): container finished" podID="5cb1cf6e-21ba-4c37-9a0b-5a076bceaf9d" containerID="248220fd6aef545cac45dabd8be83a389a483aa266822cad9d9ae0c34ce2a648" exitCode=0 Sep 10 13:21:47.422643 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:21:47.422612 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af66txf" event={"ID":"5cb1cf6e-21ba-4c37-9a0b-5a076bceaf9d","Type":"ContainerDied","Data":"248220fd6aef545cac45dabd8be83a389a483aa266822cad9d9ae0c34ce2a648"} Sep 10 13:21:48.429296 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:21:48.429256 2557 generic.go:356] "Generic (PLEG): container finished" podID="5cb1cf6e-21ba-4c37-9a0b-5a076bceaf9d" containerID="fcf45f078c406152740612d32fa99c813e71f16338ed38971361f3d4d49a1df9" exitCode=0 Sep 10 13:21:48.429736 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:21:48.429364 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af66txf" event={"ID":"5cb1cf6e-21ba-4c37-9a0b-5a076bceaf9d","Type":"ContainerDied","Data":"fcf45f078c406152740612d32fa99c813e71f16338ed38971361f3d4d49a1df9"} Sep 10 13:21:49.555890 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:21:49.555870 2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af66txf" Sep 10 13:21:49.686861 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:21:49.686787 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/5cb1cf6e-21ba-4c37-9a0b-5a076bceaf9d-bundle\") pod \"5cb1cf6e-21ba-4c37-9a0b-5a076bceaf9d\" (UID: \"5cb1cf6e-21ba-4c37-9a0b-5a076bceaf9d\") " Sep 10 13:21:49.686861 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:21:49.686847 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/5cb1cf6e-21ba-4c37-9a0b-5a076bceaf9d-util\") pod \"5cb1cf6e-21ba-4c37-9a0b-5a076bceaf9d\" (UID: \"5cb1cf6e-21ba-4c37-9a0b-5a076bceaf9d\") " Sep 10 13:21:49.687065 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:21:49.686873 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-tc69b\" (UniqueName: \"kubernetes.io/projected/5cb1cf6e-21ba-4c37-9a0b-5a076bceaf9d-kube-api-access-tc69b\") pod \"5cb1cf6e-21ba-4c37-9a0b-5a076bceaf9d\" (UID: \"5cb1cf6e-21ba-4c37-9a0b-5a076bceaf9d\") " Sep 10 13:21:49.687467 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:21:49.687439 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5cb1cf6e-21ba-4c37-9a0b-5a076bceaf9d-bundle" (OuterVolumeSpecName: "bundle") pod "5cb1cf6e-21ba-4c37-9a0b-5a076bceaf9d" (UID: "5cb1cf6e-21ba-4c37-9a0b-5a076bceaf9d"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 13:21:49.689057 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:21:49.689034 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5cb1cf6e-21ba-4c37-9a0b-5a076bceaf9d-kube-api-access-tc69b" (OuterVolumeSpecName: "kube-api-access-tc69b") pod "5cb1cf6e-21ba-4c37-9a0b-5a076bceaf9d" (UID: "5cb1cf6e-21ba-4c37-9a0b-5a076bceaf9d"). InnerVolumeSpecName "kube-api-access-tc69b". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 10 13:21:49.692399 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:21:49.692364 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5cb1cf6e-21ba-4c37-9a0b-5a076bceaf9d-util" (OuterVolumeSpecName: "util") pod "5cb1cf6e-21ba-4c37-9a0b-5a076bceaf9d" (UID: "5cb1cf6e-21ba-4c37-9a0b-5a076bceaf9d"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 13:21:49.787886 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:21:49.787857 2557 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/5cb1cf6e-21ba-4c37-9a0b-5a076bceaf9d-bundle\") on node \"ip-10-0-139-174.ec2.internal\" DevicePath \"\"" Sep 10 13:21:49.787886 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:21:49.787879 2557 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/5cb1cf6e-21ba-4c37-9a0b-5a076bceaf9d-util\") on node \"ip-10-0-139-174.ec2.internal\" DevicePath \"\"" Sep 10 13:21:49.787886 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:21:49.787888 2557 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tc69b\" (UniqueName: \"kubernetes.io/projected/5cb1cf6e-21ba-4c37-9a0b-5a076bceaf9d-kube-api-access-tc69b\") on node \"ip-10-0-139-174.ec2.internal\" DevicePath \"\"" Sep 10 13:21:50.439659 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:21:50.439629 2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af66txf" Sep 10 13:21:50.439897 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:21:50.439636 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af66txf" event={"ID":"5cb1cf6e-21ba-4c37-9a0b-5a076bceaf9d","Type":"ContainerDied","Data":"a769115a685e7cc779ea4c707a86a88424b85248d19a24edfbe9e98070875da9"} Sep 10 13:21:50.439897 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:21:50.439763 2557 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a769115a685e7cc779ea4c707a86a88424b85248d19a24edfbe9e98070875da9" Sep 10 13:21:57.517620 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:21:57.517588 2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-lws-operator/lws-controller-manager-78dbbc87c8-mf5qs"] Sep 10 13:21:57.517984 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:21:57.517912 2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5cb1cf6e-21ba-4c37-9a0b-5a076bceaf9d" containerName="extract" Sep 10 13:21:57.517984 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:21:57.517925 2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="5cb1cf6e-21ba-4c37-9a0b-5a076bceaf9d" containerName="extract" Sep 10 13:21:57.517984 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:21:57.517942 2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5cb1cf6e-21ba-4c37-9a0b-5a076bceaf9d" containerName="pull" Sep 10 13:21:57.517984 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:21:57.517953 2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="5cb1cf6e-21ba-4c37-9a0b-5a076bceaf9d" containerName="pull" Sep 10 13:21:57.517984 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:21:57.517967 2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5cb1cf6e-21ba-4c37-9a0b-5a076bceaf9d" containerName="util" Sep 10 13:21:57.517984 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:21:57.517974 2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="5cb1cf6e-21ba-4c37-9a0b-5a076bceaf9d" containerName="util" Sep 10 13:21:57.518166 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:21:57.518030 2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="5cb1cf6e-21ba-4c37-9a0b-5a076bceaf9d" containerName="extract" Sep 10 13:21:57.520166 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:21:57.520149 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-78dbbc87c8-mf5qs" Sep 10 13:21:57.525150 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:21:57.525125 2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"kube-root-ca.crt\"" Sep 10 13:21:57.525323 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:21:57.525302 2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"openshift-service-ca.crt\"" Sep 10 13:21:57.525412 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:21:57.525393 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"lws-controller-manager-dockercfg-jw76b\"" Sep 10 13:21:57.525562 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:21:57.525547 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"webhook-server-cert\"" Sep 10 13:21:57.525639 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:21:57.525592 2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"lws-manager-config\"" Sep 10 13:21:57.525694 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:21:57.525639 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"metrics-server-cert\"" Sep 10 13:21:57.538604 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:21:57.538582 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-78dbbc87c8-mf5qs"] Sep 10 13:21:57.643296 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:21:57.643267 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/7920562d-b64e-478f-b502-abb12f5f1d2a-manager-config\") pod \"lws-controller-manager-78dbbc87c8-mf5qs\" (UID: \"7920562d-b64e-478f-b502-abb12f5f1d2a\") " pod="openshift-lws-operator/lws-controller-manager-78dbbc87c8-mf5qs" Sep 10 13:21:57.643477 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:21:57.643319 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/7920562d-b64e-478f-b502-abb12f5f1d2a-cert\") pod \"lws-controller-manager-78dbbc87c8-mf5qs\" (UID: \"7920562d-b64e-478f-b502-abb12f5f1d2a\") " pod="openshift-lws-operator/lws-controller-manager-78dbbc87c8-mf5qs" Sep 10 13:21:57.643477 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:21:57.643339 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qfwxd\" (UniqueName: \"kubernetes.io/projected/7920562d-b64e-478f-b502-abb12f5f1d2a-kube-api-access-qfwxd\") pod \"lws-controller-manager-78dbbc87c8-mf5qs\" (UID: \"7920562d-b64e-478f-b502-abb12f5f1d2a\") " pod="openshift-lws-operator/lws-controller-manager-78dbbc87c8-mf5qs" Sep 10 13:21:57.643477 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:21:57.643373 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/7920562d-b64e-478f-b502-abb12f5f1d2a-metrics-cert\") pod \"lws-controller-manager-78dbbc87c8-mf5qs\" (UID: \"7920562d-b64e-478f-b502-abb12f5f1d2a\") " pod="openshift-lws-operator/lws-controller-manager-78dbbc87c8-mf5qs" Sep 10 13:21:57.744067 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:21:57.744037 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/7920562d-b64e-478f-b502-abb12f5f1d2a-metrics-cert\") pod \"lws-controller-manager-78dbbc87c8-mf5qs\" (UID: \"7920562d-b64e-478f-b502-abb12f5f1d2a\") " pod="openshift-lws-operator/lws-controller-manager-78dbbc87c8-mf5qs" Sep 10 13:21:57.744223 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:21:57.744081 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/7920562d-b64e-478f-b502-abb12f5f1d2a-manager-config\") pod \"lws-controller-manager-78dbbc87c8-mf5qs\" (UID: \"7920562d-b64e-478f-b502-abb12f5f1d2a\") " pod="openshift-lws-operator/lws-controller-manager-78dbbc87c8-mf5qs" Sep 10 13:21:57.744223 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:21:57.744129 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/7920562d-b64e-478f-b502-abb12f5f1d2a-cert\") pod \"lws-controller-manager-78dbbc87c8-mf5qs\" (UID: \"7920562d-b64e-478f-b502-abb12f5f1d2a\") " pod="openshift-lws-operator/lws-controller-manager-78dbbc87c8-mf5qs" Sep 10 13:21:57.744223 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:21:57.744148 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qfwxd\" (UniqueName: \"kubernetes.io/projected/7920562d-b64e-478f-b502-abb12f5f1d2a-kube-api-access-qfwxd\") pod \"lws-controller-manager-78dbbc87c8-mf5qs\" (UID: \"7920562d-b64e-478f-b502-abb12f5f1d2a\") " pod="openshift-lws-operator/lws-controller-manager-78dbbc87c8-mf5qs" Sep 10 13:21:57.744819 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:21:57.744794 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/7920562d-b64e-478f-b502-abb12f5f1d2a-manager-config\") pod \"lws-controller-manager-78dbbc87c8-mf5qs\" (UID: \"7920562d-b64e-478f-b502-abb12f5f1d2a\") " pod="openshift-lws-operator/lws-controller-manager-78dbbc87c8-mf5qs" Sep 10 13:21:57.746631 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:21:57.746611 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/7920562d-b64e-478f-b502-abb12f5f1d2a-cert\") pod \"lws-controller-manager-78dbbc87c8-mf5qs\" (UID: \"7920562d-b64e-478f-b502-abb12f5f1d2a\") " pod="openshift-lws-operator/lws-controller-manager-78dbbc87c8-mf5qs" Sep 10 13:21:57.746707 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:21:57.746656 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/7920562d-b64e-478f-b502-abb12f5f1d2a-metrics-cert\") pod \"lws-controller-manager-78dbbc87c8-mf5qs\" (UID: \"7920562d-b64e-478f-b502-abb12f5f1d2a\") " pod="openshift-lws-operator/lws-controller-manager-78dbbc87c8-mf5qs" Sep 10 13:21:57.761996 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:21:57.761977 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qfwxd\" (UniqueName: \"kubernetes.io/projected/7920562d-b64e-478f-b502-abb12f5f1d2a-kube-api-access-qfwxd\") pod \"lws-controller-manager-78dbbc87c8-mf5qs\" (UID: \"7920562d-b64e-478f-b502-abb12f5f1d2a\") " pod="openshift-lws-operator/lws-controller-manager-78dbbc87c8-mf5qs" Sep 10 13:21:57.830042 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:21:57.829976 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-78dbbc87c8-mf5qs" Sep 10 13:21:58.141329 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:21:58.141164 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-78dbbc87c8-mf5qs"] Sep 10 13:21:58.143846 ip-10-0-139-174 kubenswrapper[2557]: W0910 13:21:58.143819 2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7920562d_b64e_478f_b502_abb12f5f1d2a.slice/crio-cf10a071fd48025eb47de750f7cd854a07562f6c5edb1f5adcf697e965f93fc7 WatchSource:0}: Error finding container cf10a071fd48025eb47de750f7cd854a07562f6c5edb1f5adcf697e965f93fc7: Status 404 returned error can't find the container with id cf10a071fd48025eb47de750f7cd854a07562f6c5edb1f5adcf697e965f93fc7 Sep 10 13:21:58.476476 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:21:58.476443 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-78dbbc87c8-mf5qs" event={"ID":"7920562d-b64e-478f-b502-abb12f5f1d2a","Type":"ContainerStarted","Data":"cf10a071fd48025eb47de750f7cd854a07562f6c5edb1f5adcf697e965f93fc7"} Sep 10 13:21:59.128028 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:21:59.127991 2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0g48fx"] Sep 10 13:21:59.130965 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:21:59.130943 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0g48fx" Sep 10 13:21:59.134407 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:21:59.134375 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-9r2bh\"" Sep 10 13:21:59.134900 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:21:59.134572 2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\"" Sep 10 13:21:59.134900 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:21:59.134753 2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\"" Sep 10 13:21:59.155367 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:21:59.155304 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0g48fx"] Sep 10 13:21:59.258371 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:21:59.258333 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/08644ca9-e6a5-40c4-8c33-eed13bc355fb-util\") pod \"8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0g48fx\" (UID: \"08644ca9-e6a5-40c4-8c33-eed13bc355fb\") " pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0g48fx" Sep 10 13:21:59.258563 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:21:59.258481 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/08644ca9-e6a5-40c4-8c33-eed13bc355fb-bundle\") pod \"8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0g48fx\" (UID: \"08644ca9-e6a5-40c4-8c33-eed13bc355fb\") " pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0g48fx" Sep 10 13:21:59.258563 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:21:59.258516 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qppgp\" (UniqueName: \"kubernetes.io/projected/08644ca9-e6a5-40c4-8c33-eed13bc355fb-kube-api-access-qppgp\") pod \"8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0g48fx\" (UID: \"08644ca9-e6a5-40c4-8c33-eed13bc355fb\") " pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0g48fx" Sep 10 13:21:59.359890 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:21:59.359851 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/08644ca9-e6a5-40c4-8c33-eed13bc355fb-bundle\") pod \"8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0g48fx\" (UID: \"08644ca9-e6a5-40c4-8c33-eed13bc355fb\") " pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0g48fx" Sep 10 13:21:59.360078 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:21:59.359901 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qppgp\" (UniqueName: \"kubernetes.io/projected/08644ca9-e6a5-40c4-8c33-eed13bc355fb-kube-api-access-qppgp\") pod \"8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0g48fx\" (UID: \"08644ca9-e6a5-40c4-8c33-eed13bc355fb\") " pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0g48fx" Sep 10 13:21:59.360078 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:21:59.359969 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/08644ca9-e6a5-40c4-8c33-eed13bc355fb-util\") pod \"8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0g48fx\" (UID: \"08644ca9-e6a5-40c4-8c33-eed13bc355fb\") " pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0g48fx" Sep 10 13:21:59.360268 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:21:59.360237 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/08644ca9-e6a5-40c4-8c33-eed13bc355fb-bundle\") pod \"8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0g48fx\" (UID: \"08644ca9-e6a5-40c4-8c33-eed13bc355fb\") " pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0g48fx" Sep 10 13:21:59.360402 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:21:59.360317 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/08644ca9-e6a5-40c4-8c33-eed13bc355fb-util\") pod \"8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0g48fx\" (UID: \"08644ca9-e6a5-40c4-8c33-eed13bc355fb\") " pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0g48fx" Sep 10 13:21:59.376452 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:21:59.376408 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qppgp\" (UniqueName: \"kubernetes.io/projected/08644ca9-e6a5-40c4-8c33-eed13bc355fb-kube-api-access-qppgp\") pod \"8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0g48fx\" (UID: \"08644ca9-e6a5-40c4-8c33-eed13bc355fb\") " pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0g48fx" Sep 10 13:21:59.443224 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:21:59.443160 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0g48fx" Sep 10 13:21:59.820914 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:21:59.820885 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0g48fx"] Sep 10 13:21:59.823932 ip-10-0-139-174 kubenswrapper[2557]: W0910 13:21:59.823907 2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod08644ca9_e6a5_40c4_8c33_eed13bc355fb.slice/crio-4ac7c23767112efb61c9dfbd9f3e17e503d3b2fe9688af455b6a81f4d93b08d5 WatchSource:0}: Error finding container 4ac7c23767112efb61c9dfbd9f3e17e503d3b2fe9688af455b6a81f4d93b08d5: Status 404 returned error can't find the container with id 4ac7c23767112efb61c9dfbd9f3e17e503d3b2fe9688af455b6a81f4d93b08d5 Sep 10 13:22:00.488781 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:22:00.488742 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-78dbbc87c8-mf5qs" event={"ID":"7920562d-b64e-478f-b502-abb12f5f1d2a","Type":"ContainerStarted","Data":"1c7fc1f248c583c1def7ae5a3bab8ffe790547bfb14c013efc9a5205d783edf2"} Sep 10 13:22:00.489242 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:22:00.488815 2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-lws-operator/lws-controller-manager-78dbbc87c8-mf5qs" Sep 10 13:22:00.490233 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:22:00.490204 2557 generic.go:356] "Generic (PLEG): container finished" podID="08644ca9-e6a5-40c4-8c33-eed13bc355fb" containerID="27b86feb7fc3d610f9db808cece6888fbf01ddb5bdf6f0ca51b9f67e91c3fd72" exitCode=0 Sep 10 13:22:00.490378 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:22:00.490253 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0g48fx" event={"ID":"08644ca9-e6a5-40c4-8c33-eed13bc355fb","Type":"ContainerDied","Data":"27b86feb7fc3d610f9db808cece6888fbf01ddb5bdf6f0ca51b9f67e91c3fd72"} Sep 10 13:22:00.490378 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:22:00.490296 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0g48fx" event={"ID":"08644ca9-e6a5-40c4-8c33-eed13bc355fb","Type":"ContainerStarted","Data":"4ac7c23767112efb61c9dfbd9f3e17e503d3b2fe9688af455b6a81f4d93b08d5"} Sep 10 13:22:00.522686 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:22:00.522635 2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-lws-operator/lws-controller-manager-78dbbc87c8-mf5qs" podStartSLOduration=2.095742286 podStartE2EDuration="3.522621681s" totalImagesPullingTime="1.426879395s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 13:21:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 13:22:00.521413216 +0000 UTC m=+280.828407631" watchObservedRunningTime="2026-09-10 13:22:00.522621681 +0000 UTC m=+280.829616094" Sep 10 13:22:01.496151 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:22:01.496120 2557 generic.go:356] "Generic (PLEG): container finished" podID="08644ca9-e6a5-40c4-8c33-eed13bc355fb" containerID="dcac0e355f3223ace15cfadf8d20bd6455bb218df26a7fb30fbc4824c7f13832" exitCode=0 Sep 10 13:22:01.496578 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:22:01.496209 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0g48fx" event={"ID":"08644ca9-e6a5-40c4-8c33-eed13bc355fb","Type":"ContainerDied","Data":"dcac0e355f3223ace15cfadf8d20bd6455bb218df26a7fb30fbc4824c7f13832"} Sep 10 13:22:02.258410 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:22:02.258379 2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-699db476d-5nldv"] Sep 10 13:22:02.260691 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:22:02.260673 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-699db476d-5nldv" Sep 10 13:22:02.266213 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:22:02.266192 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-webhook-cert\"" Sep 10 13:22:02.266378 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:22:02.266360 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-service-cert\"" Sep 10 13:22:02.266519 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:22:02.266502 2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\"" Sep 10 13:22:02.266582 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:22:02.266566 2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\"" Sep 10 13:22:02.267317 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:22:02.267295 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-dockercfg-lrcqt\"" Sep 10 13:22:02.316086 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:22:02.316059 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-699db476d-5nldv"] Sep 10 13:22:02.388530 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:22:02.388500 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/ba4fa399-bbb1-4d97-9b2c-4d19667904c4-webhook-cert\") pod \"opendatahub-operator-controller-manager-699db476d-5nldv\" (UID: \"ba4fa399-bbb1-4d97-9b2c-4d19667904c4\") " pod="opendatahub/opendatahub-operator-controller-manager-699db476d-5nldv" Sep 10 13:22:02.388763 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:22:02.388545 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/ba4fa399-bbb1-4d97-9b2c-4d19667904c4-tmp\") pod \"opendatahub-operator-controller-manager-699db476d-5nldv\" (UID: \"ba4fa399-bbb1-4d97-9b2c-4d19667904c4\") " pod="opendatahub/opendatahub-operator-controller-manager-699db476d-5nldv" Sep 10 13:22:02.388763 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:22:02.388566 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/ba4fa399-bbb1-4d97-9b2c-4d19667904c4-apiservice-cert\") pod \"opendatahub-operator-controller-manager-699db476d-5nldv\" (UID: \"ba4fa399-bbb1-4d97-9b2c-4d19667904c4\") " pod="opendatahub/opendatahub-operator-controller-manager-699db476d-5nldv" Sep 10 13:22:02.388763 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:22:02.388590 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z572w\" (UniqueName: \"kubernetes.io/projected/ba4fa399-bbb1-4d97-9b2c-4d19667904c4-kube-api-access-z572w\") pod \"opendatahub-operator-controller-manager-699db476d-5nldv\" (UID: \"ba4fa399-bbb1-4d97-9b2c-4d19667904c4\") " pod="opendatahub/opendatahub-operator-controller-manager-699db476d-5nldv" Sep 10 13:22:02.388763 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:22:02.388666 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/ba4fa399-bbb1-4d97-9b2c-4d19667904c4-manifests\") pod \"opendatahub-operator-controller-manager-699db476d-5nldv\" (UID: \"ba4fa399-bbb1-4d97-9b2c-4d19667904c4\") " pod="opendatahub/opendatahub-operator-controller-manager-699db476d-5nldv" Sep 10 13:22:02.489339 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:22:02.489303 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/ba4fa399-bbb1-4d97-9b2c-4d19667904c4-tmp\") pod \"opendatahub-operator-controller-manager-699db476d-5nldv\" (UID: \"ba4fa399-bbb1-4d97-9b2c-4d19667904c4\") " pod="opendatahub/opendatahub-operator-controller-manager-699db476d-5nldv" Sep 10 13:22:02.489500 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:22:02.489342 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/ba4fa399-bbb1-4d97-9b2c-4d19667904c4-apiservice-cert\") pod \"opendatahub-operator-controller-manager-699db476d-5nldv\" (UID: \"ba4fa399-bbb1-4d97-9b2c-4d19667904c4\") " pod="opendatahub/opendatahub-operator-controller-manager-699db476d-5nldv" Sep 10 13:22:02.489500 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:22:02.489471 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z572w\" (UniqueName: \"kubernetes.io/projected/ba4fa399-bbb1-4d97-9b2c-4d19667904c4-kube-api-access-z572w\") pod \"opendatahub-operator-controller-manager-699db476d-5nldv\" (UID: \"ba4fa399-bbb1-4d97-9b2c-4d19667904c4\") " pod="opendatahub/opendatahub-operator-controller-manager-699db476d-5nldv" Sep 10 13:22:02.489605 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:22:02.489530 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/ba4fa399-bbb1-4d97-9b2c-4d19667904c4-manifests\") pod \"opendatahub-operator-controller-manager-699db476d-5nldv\" (UID: \"ba4fa399-bbb1-4d97-9b2c-4d19667904c4\") " pod="opendatahub/opendatahub-operator-controller-manager-699db476d-5nldv" Sep 10 13:22:02.489662 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:22:02.489608 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/ba4fa399-bbb1-4d97-9b2c-4d19667904c4-webhook-cert\") pod \"opendatahub-operator-controller-manager-699db476d-5nldv\" (UID: \"ba4fa399-bbb1-4d97-9b2c-4d19667904c4\") " pod="opendatahub/opendatahub-operator-controller-manager-699db476d-5nldv" Sep 10 13:22:02.489883 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:22:02.489857 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/ba4fa399-bbb1-4d97-9b2c-4d19667904c4-manifests\") pod \"opendatahub-operator-controller-manager-699db476d-5nldv\" (UID: \"ba4fa399-bbb1-4d97-9b2c-4d19667904c4\") " pod="opendatahub/opendatahub-operator-controller-manager-699db476d-5nldv" Sep 10 13:22:02.491817 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:22:02.491794 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/ba4fa399-bbb1-4d97-9b2c-4d19667904c4-tmp\") pod \"opendatahub-operator-controller-manager-699db476d-5nldv\" (UID: \"ba4fa399-bbb1-4d97-9b2c-4d19667904c4\") " pod="opendatahub/opendatahub-operator-controller-manager-699db476d-5nldv" Sep 10 13:22:02.491974 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:22:02.491959 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/ba4fa399-bbb1-4d97-9b2c-4d19667904c4-webhook-cert\") pod \"opendatahub-operator-controller-manager-699db476d-5nldv\" (UID: \"ba4fa399-bbb1-4d97-9b2c-4d19667904c4\") " pod="opendatahub/opendatahub-operator-controller-manager-699db476d-5nldv" Sep 10 13:22:02.492022 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:22:02.491994 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/ba4fa399-bbb1-4d97-9b2c-4d19667904c4-apiservice-cert\") pod \"opendatahub-operator-controller-manager-699db476d-5nldv\" (UID: \"ba4fa399-bbb1-4d97-9b2c-4d19667904c4\") " pod="opendatahub/opendatahub-operator-controller-manager-699db476d-5nldv" Sep 10 13:22:02.503249 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:22:02.503228 2557 generic.go:356] "Generic (PLEG): container finished" podID="08644ca9-e6a5-40c4-8c33-eed13bc355fb" containerID="fb9700471a25fabcc6adbc2f62fae547d37f23ad2bcab54d56fbeca3d7b7ced4" exitCode=0 Sep 10 13:22:02.503556 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:22:02.503290 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0g48fx" event={"ID":"08644ca9-e6a5-40c4-8c33-eed13bc355fb","Type":"ContainerDied","Data":"fb9700471a25fabcc6adbc2f62fae547d37f23ad2bcab54d56fbeca3d7b7ced4"} Sep 10 13:22:02.510479 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:22:02.510411 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-z572w\" (UniqueName: \"kubernetes.io/projected/ba4fa399-bbb1-4d97-9b2c-4d19667904c4-kube-api-access-z572w\") pod \"opendatahub-operator-controller-manager-699db476d-5nldv\" (UID: \"ba4fa399-bbb1-4d97-9b2c-4d19667904c4\") " pod="opendatahub/opendatahub-operator-controller-manager-699db476d-5nldv" Sep 10 13:22:02.570342 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:22:02.570312 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-699db476d-5nldv" Sep 10 13:22:02.869584 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:22:02.869509 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-699db476d-5nldv"] Sep 10 13:22:02.872864 ip-10-0-139-174 kubenswrapper[2557]: W0910 13:22:02.872838 2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podba4fa399_bbb1_4d97_9b2c_4d19667904c4.slice/crio-61ba37bedd2b2ef65c13f2620a1923c1159e83ec8141c0cde5fbbb002d8f549a WatchSource:0}: Error finding container 61ba37bedd2b2ef65c13f2620a1923c1159e83ec8141c0cde5fbbb002d8f549a: Status 404 returned error can't find the container with id 61ba37bedd2b2ef65c13f2620a1923c1159e83ec8141c0cde5fbbb002d8f549a Sep 10 13:22:03.515959 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:22:03.515916 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-699db476d-5nldv" event={"ID":"ba4fa399-bbb1-4d97-9b2c-4d19667904c4","Type":"ContainerStarted","Data":"61ba37bedd2b2ef65c13f2620a1923c1159e83ec8141c0cde5fbbb002d8f549a"} Sep 10 13:22:03.682146 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:22:03.682029 2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0g48fx" Sep 10 13:22:03.799720 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:22:03.799640 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/08644ca9-e6a5-40c4-8c33-eed13bc355fb-util\") pod \"08644ca9-e6a5-40c4-8c33-eed13bc355fb\" (UID: \"08644ca9-e6a5-40c4-8c33-eed13bc355fb\") " Sep 10 13:22:03.799720 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:22:03.799704 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qppgp\" (UniqueName: \"kubernetes.io/projected/08644ca9-e6a5-40c4-8c33-eed13bc355fb-kube-api-access-qppgp\") pod \"08644ca9-e6a5-40c4-8c33-eed13bc355fb\" (UID: \"08644ca9-e6a5-40c4-8c33-eed13bc355fb\") " Sep 10 13:22:03.799938 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:22:03.799765 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/08644ca9-e6a5-40c4-8c33-eed13bc355fb-bundle\") pod \"08644ca9-e6a5-40c4-8c33-eed13bc355fb\" (UID: \"08644ca9-e6a5-40c4-8c33-eed13bc355fb\") " Sep 10 13:22:03.800939 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:22:03.800892 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/08644ca9-e6a5-40c4-8c33-eed13bc355fb-bundle" (OuterVolumeSpecName: "bundle") pod "08644ca9-e6a5-40c4-8c33-eed13bc355fb" (UID: "08644ca9-e6a5-40c4-8c33-eed13bc355fb"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 13:22:03.802507 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:22:03.802453 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/08644ca9-e6a5-40c4-8c33-eed13bc355fb-kube-api-access-qppgp" (OuterVolumeSpecName: "kube-api-access-qppgp") pod "08644ca9-e6a5-40c4-8c33-eed13bc355fb" (UID: "08644ca9-e6a5-40c4-8c33-eed13bc355fb"). InnerVolumeSpecName "kube-api-access-qppgp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 10 13:22:03.807814 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:22:03.807779 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/08644ca9-e6a5-40c4-8c33-eed13bc355fb-util" (OuterVolumeSpecName: "util") pod "08644ca9-e6a5-40c4-8c33-eed13bc355fb" (UID: "08644ca9-e6a5-40c4-8c33-eed13bc355fb"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 13:22:03.900724 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:22:03.900664 2557 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/08644ca9-e6a5-40c4-8c33-eed13bc355fb-util\") on node \"ip-10-0-139-174.ec2.internal\" DevicePath \"\"" Sep 10 13:22:03.900724 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:22:03.900724 2557 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qppgp\" (UniqueName: \"kubernetes.io/projected/08644ca9-e6a5-40c4-8c33-eed13bc355fb-kube-api-access-qppgp\") on node \"ip-10-0-139-174.ec2.internal\" DevicePath \"\"" Sep 10 13:22:03.900941 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:22:03.900740 2557 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/08644ca9-e6a5-40c4-8c33-eed13bc355fb-bundle\") on node \"ip-10-0-139-174.ec2.internal\" DevicePath \"\"" Sep 10 13:22:04.525105 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:22:04.525025 2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0g48fx" Sep 10 13:22:04.525105 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:22:04.525026 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0g48fx" event={"ID":"08644ca9-e6a5-40c4-8c33-eed13bc355fb","Type":"ContainerDied","Data":"4ac7c23767112efb61c9dfbd9f3e17e503d3b2fe9688af455b6a81f4d93b08d5"} Sep 10 13:22:04.525105 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:22:04.525090 2557 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="4ac7c23767112efb61c9dfbd9f3e17e503d3b2fe9688af455b6a81f4d93b08d5" Sep 10 13:22:06.537462 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:22:06.537407 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-699db476d-5nldv" event={"ID":"ba4fa399-bbb1-4d97-9b2c-4d19667904c4","Type":"ContainerStarted","Data":"926fcba9d616daf476ff64bef4fc451e96785b895553df0a3aafaa9a5477479d"} Sep 10 13:22:07.544251 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:22:07.544212 2557 generic.go:356] "Generic (PLEG): container finished" podID="ba4fa399-bbb1-4d97-9b2c-4d19667904c4" containerID="926fcba9d616daf476ff64bef4fc451e96785b895553df0a3aafaa9a5477479d" exitCode=0 Sep 10 13:22:07.544661 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:22:07.544307 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-699db476d-5nldv" event={"ID":"ba4fa399-bbb1-4d97-9b2c-4d19667904c4","Type":"ContainerDied","Data":"926fcba9d616daf476ff64bef4fc451e96785b895553df0a3aafaa9a5477479d"} Sep 10 13:22:08.550643 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:22:08.550610 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-699db476d-5nldv" event={"ID":"ba4fa399-bbb1-4d97-9b2c-4d19667904c4","Type":"ContainerStarted","Data":"2fa405388099cfb30ef5552158c0780a40d08940043a4a0320b3663a213184f9"} Sep 10 13:22:08.551026 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:22:08.550776 2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/opendatahub-operator-controller-manager-699db476d-5nldv" Sep 10 13:22:08.606653 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:22:08.606604 2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/opendatahub-operator-controller-manager-699db476d-5nldv" podStartSLOduration=2.206657794 podStartE2EDuration="6.6065897s" totalImagesPullingTime="3.366297029s" totalInitContainerRuntime="1.033634877s" isStatefulPod=false podCreationTimestamp="2026-09-10 13:22:02 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 13:22:08.605655358 +0000 UTC m=+288.912649771" watchObservedRunningTime="2026-09-10 13:22:08.6065897 +0000 UTC m=+288.913584114" Sep 10 13:22:11.498950 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:22:11.498923 2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-lws-operator/lws-controller-manager-78dbbc87c8-mf5qs" Sep 10 13:22:19.557542 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:22:19.557511 2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/opendatahub-operator-controller-manager-699db476d-5nldv" Sep 10 13:22:20.098484 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:22:20.098445 2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-92n4q_720acda6-0b15-49ac-ab4e-60de70c4324a/ovn-acl-logging/0.log" Sep 10 13:22:20.100554 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:22:20.100528 2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-92n4q_720acda6-0b15-49ac-ab4e-60de70c4324a/ovn-acl-logging/0.log" Sep 10 13:22:20.101490 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:22:20.101470 2557 kubelet.go:1648] "Image garbage collection succeeded" Sep 10 13:22:33.308669 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:22:33.308636 2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-fddd6c88c-gf8ms"] Sep 10 13:22:33.309065 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:22:33.308916 2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="08644ca9-e6a5-40c4-8c33-eed13bc355fb" containerName="extract" Sep 10 13:22:33.309065 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:22:33.308927 2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="08644ca9-e6a5-40c4-8c33-eed13bc355fb" containerName="extract" Sep 10 13:22:33.309065 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:22:33.308939 2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="08644ca9-e6a5-40c4-8c33-eed13bc355fb" containerName="util" Sep 10 13:22:33.309065 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:22:33.308944 2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="08644ca9-e6a5-40c4-8c33-eed13bc355fb" containerName="util" Sep 10 13:22:33.309065 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:22:33.308960 2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="08644ca9-e6a5-40c4-8c33-eed13bc355fb" containerName="pull" Sep 10 13:22:33.309065 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:22:33.308965 2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="08644ca9-e6a5-40c4-8c33-eed13bc355fb" containerName="pull" Sep 10 13:22:33.309065 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:22:33.309015 2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="08644ca9-e6a5-40c4-8c33-eed13bc355fb" containerName="extract" Sep 10 13:22:33.311212 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:22:33.311190 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-fddd6c88c-gf8ms" Sep 10 13:22:33.316982 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:22:33.316963 2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\"" Sep 10 13:22:33.317091 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:22:33.317001 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"data-science-gateway-data-science-gateway-class-dockercfg-wm2p8\"" Sep 10 13:22:33.317471 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:22:33.317454 2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\"" Sep 10 13:22:33.317574 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:22:33.317527 2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\"" Sep 10 13:22:33.341236 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:22:33.341191 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-fddd6c88c-gf8ms"] Sep 10 13:22:33.411639 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:22:33.411612 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/30d85e7c-08a7-41a2-beb0-4a5b33c2f826-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-fddd6c88c-gf8ms\" (UID: \"30d85e7c-08a7-41a2-beb0-4a5b33c2f826\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-fddd6c88c-gf8ms" Sep 10 13:22:33.411754 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:22:33.411642 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/30d85e7c-08a7-41a2-beb0-4a5b33c2f826-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-fddd6c88c-gf8ms\" (UID: \"30d85e7c-08a7-41a2-beb0-4a5b33c2f826\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-fddd6c88c-gf8ms" Sep 10 13:22:33.411754 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:22:33.411666 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/30d85e7c-08a7-41a2-beb0-4a5b33c2f826-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-fddd6c88c-gf8ms\" (UID: \"30d85e7c-08a7-41a2-beb0-4a5b33c2f826\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-fddd6c88c-gf8ms" Sep 10 13:22:33.411754 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:22:33.411681 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/30d85e7c-08a7-41a2-beb0-4a5b33c2f826-istio-token\") pod \"data-science-gateway-data-science-gateway-class-fddd6c88c-gf8ms\" (UID: \"30d85e7c-08a7-41a2-beb0-4a5b33c2f826\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-fddd6c88c-gf8ms" Sep 10 13:22:33.411754 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:22:33.411698 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/30d85e7c-08a7-41a2-beb0-4a5b33c2f826-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-fddd6c88c-gf8ms\" (UID: \"30d85e7c-08a7-41a2-beb0-4a5b33c2f826\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-fddd6c88c-gf8ms" Sep 10 13:22:33.411754 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:22:33.411731 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hv5dl\" (UniqueName: \"kubernetes.io/projected/30d85e7c-08a7-41a2-beb0-4a5b33c2f826-kube-api-access-hv5dl\") pod \"data-science-gateway-data-science-gateway-class-fddd6c88c-gf8ms\" (UID: \"30d85e7c-08a7-41a2-beb0-4a5b33c2f826\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-fddd6c88c-gf8ms" Sep 10 13:22:33.411754 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:22:33.411748 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/30d85e7c-08a7-41a2-beb0-4a5b33c2f826-istio-data\") pod \"data-science-gateway-data-science-gateway-class-fddd6c88c-gf8ms\" (UID: \"30d85e7c-08a7-41a2-beb0-4a5b33c2f826\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-fddd6c88c-gf8ms" Sep 10 13:22:33.412031 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:22:33.411821 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/30d85e7c-08a7-41a2-beb0-4a5b33c2f826-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-fddd6c88c-gf8ms\" (UID: \"30d85e7c-08a7-41a2-beb0-4a5b33c2f826\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-fddd6c88c-gf8ms" Sep 10 13:22:33.412031 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:22:33.411891 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/30d85e7c-08a7-41a2-beb0-4a5b33c2f826-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-fddd6c88c-gf8ms\" (UID: \"30d85e7c-08a7-41a2-beb0-4a5b33c2f826\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-fddd6c88c-gf8ms" Sep 10 13:22:33.497566 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:22:33.497537 2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799cnxbst"] Sep 10 13:22:33.500198 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:22:33.500177 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799cnxbst" Sep 10 13:22:33.503722 ip-10-0-139-174 kubenswrapper[2557]: E0910 13:22:33.503695 2557 status_manager.go:1018] "Failed to get status for pod" err="pods \"data-science-gateway-data-science-gateway-class-68cf4d799cnxbst\" is forbidden: User \"system:node:ip-10-0-139-174.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"openshift-ingress\": no relationship found between node 'ip-10-0-139-174.ec2.internal' and this object" podUID="4e2d7c7f-f21c-41ec-aaf6-15d3976f0888" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799cnxbst" Sep 10 13:22:33.512211 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:22:33.512187 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/30d85e7c-08a7-41a2-beb0-4a5b33c2f826-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-fddd6c88c-gf8ms\" (UID: \"30d85e7c-08a7-41a2-beb0-4a5b33c2f826\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-fddd6c88c-gf8ms" Sep 10 13:22:33.512316 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:22:33.512227 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/30d85e7c-08a7-41a2-beb0-4a5b33c2f826-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-fddd6c88c-gf8ms\" (UID: \"30d85e7c-08a7-41a2-beb0-4a5b33c2f826\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-fddd6c88c-gf8ms" Sep 10 13:22:33.512316 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:22:33.512264 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/30d85e7c-08a7-41a2-beb0-4a5b33c2f826-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-fddd6c88c-gf8ms\" (UID: \"30d85e7c-08a7-41a2-beb0-4a5b33c2f826\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-fddd6c88c-gf8ms" Sep 10 13:22:33.512316 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:22:33.512288 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/30d85e7c-08a7-41a2-beb0-4a5b33c2f826-istio-token\") pod \"data-science-gateway-data-science-gateway-class-fddd6c88c-gf8ms\" (UID: \"30d85e7c-08a7-41a2-beb0-4a5b33c2f826\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-fddd6c88c-gf8ms" Sep 10 13:22:33.512545 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:22:33.512314 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/30d85e7c-08a7-41a2-beb0-4a5b33c2f826-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-fddd6c88c-gf8ms\" (UID: \"30d85e7c-08a7-41a2-beb0-4a5b33c2f826\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-fddd6c88c-gf8ms" Sep 10 13:22:33.512545 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:22:33.512347 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hv5dl\" (UniqueName: \"kubernetes.io/projected/30d85e7c-08a7-41a2-beb0-4a5b33c2f826-kube-api-access-hv5dl\") pod \"data-science-gateway-data-science-gateway-class-fddd6c88c-gf8ms\" (UID: \"30d85e7c-08a7-41a2-beb0-4a5b33c2f826\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-fddd6c88c-gf8ms" Sep 10 13:22:33.512545 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:22:33.512378 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/30d85e7c-08a7-41a2-beb0-4a5b33c2f826-istio-data\") pod \"data-science-gateway-data-science-gateway-class-fddd6c88c-gf8ms\" (UID: \"30d85e7c-08a7-41a2-beb0-4a5b33c2f826\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-fddd6c88c-gf8ms" Sep 10 13:22:33.512545 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:22:33.512442 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/30d85e7c-08a7-41a2-beb0-4a5b33c2f826-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-fddd6c88c-gf8ms\" (UID: \"30d85e7c-08a7-41a2-beb0-4a5b33c2f826\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-fddd6c88c-gf8ms" Sep 10 13:22:33.512545 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:22:33.512494 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/30d85e7c-08a7-41a2-beb0-4a5b33c2f826-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-fddd6c88c-gf8ms\" (UID: \"30d85e7c-08a7-41a2-beb0-4a5b33c2f826\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-fddd6c88c-gf8ms" Sep 10 13:22:33.512801 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:22:33.512649 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/30d85e7c-08a7-41a2-beb0-4a5b33c2f826-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-fddd6c88c-gf8ms\" (UID: \"30d85e7c-08a7-41a2-beb0-4a5b33c2f826\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-fddd6c88c-gf8ms" Sep 10 13:22:33.512853 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:22:33.512803 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/30d85e7c-08a7-41a2-beb0-4a5b33c2f826-istio-data\") pod \"data-science-gateway-data-science-gateway-class-fddd6c88c-gf8ms\" (UID: \"30d85e7c-08a7-41a2-beb0-4a5b33c2f826\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-fddd6c88c-gf8ms" Sep 10 13:22:33.512853 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:22:33.512802 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/30d85e7c-08a7-41a2-beb0-4a5b33c2f826-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-fddd6c88c-gf8ms\" (UID: \"30d85e7c-08a7-41a2-beb0-4a5b33c2f826\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-fddd6c88c-gf8ms" Sep 10 13:22:33.512933 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:22:33.512898 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/30d85e7c-08a7-41a2-beb0-4a5b33c2f826-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-fddd6c88c-gf8ms\" (UID: \"30d85e7c-08a7-41a2-beb0-4a5b33c2f826\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-fddd6c88c-gf8ms" Sep 10 13:22:33.512970 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:22:33.512933 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/30d85e7c-08a7-41a2-beb0-4a5b33c2f826-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-fddd6c88c-gf8ms\" (UID: \"30d85e7c-08a7-41a2-beb0-4a5b33c2f826\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-fddd6c88c-gf8ms" Sep 10 13:22:33.515194 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:22:33.515171 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/30d85e7c-08a7-41a2-beb0-4a5b33c2f826-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-fddd6c88c-gf8ms\" (UID: \"30d85e7c-08a7-41a2-beb0-4a5b33c2f826\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-fddd6c88c-gf8ms" Sep 10 13:22:33.515359 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:22:33.515340 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/30d85e7c-08a7-41a2-beb0-4a5b33c2f826-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-fddd6c88c-gf8ms\" (UID: \"30d85e7c-08a7-41a2-beb0-4a5b33c2f826\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-fddd6c88c-gf8ms" Sep 10 13:22:33.538605 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:22:33.538573 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799cnxbst"] Sep 10 13:22:33.543844 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:22:33.543818 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hv5dl\" (UniqueName: \"kubernetes.io/projected/30d85e7c-08a7-41a2-beb0-4a5b33c2f826-kube-api-access-hv5dl\") pod \"data-science-gateway-data-science-gateway-class-fddd6c88c-gf8ms\" (UID: \"30d85e7c-08a7-41a2-beb0-4a5b33c2f826\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-fddd6c88c-gf8ms" Sep 10 13:22:33.543942 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:22:33.543856 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/30d85e7c-08a7-41a2-beb0-4a5b33c2f826-istio-token\") pod \"data-science-gateway-data-science-gateway-class-fddd6c88c-gf8ms\" (UID: \"30d85e7c-08a7-41a2-beb0-4a5b33c2f826\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-fddd6c88c-gf8ms" Sep 10 13:22:33.613594 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:22:33.613531 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4e2d7c7f-f21c-41ec-aaf6-15d3976f0888-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-68cf4d799cnxbst\" (UID: \"4e2d7c7f-f21c-41ec-aaf6-15d3976f0888\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799cnxbst" Sep 10 13:22:33.613594 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:22:33.613565 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4e2d7c7f-f21c-41ec-aaf6-15d3976f0888-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-68cf4d799cnxbst\" (UID: \"4e2d7c7f-f21c-41ec-aaf6-15d3976f0888\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799cnxbst" Sep 10 13:22:33.613594 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:22:33.613584 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-47l76\" (UniqueName: \"kubernetes.io/projected/4e2d7c7f-f21c-41ec-aaf6-15d3976f0888-kube-api-access-47l76\") pod \"data-science-gateway-data-science-gateway-class-68cf4d799cnxbst\" (UID: \"4e2d7c7f-f21c-41ec-aaf6-15d3976f0888\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799cnxbst" Sep 10 13:22:33.613758 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:22:33.613611 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4e2d7c7f-f21c-41ec-aaf6-15d3976f0888-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-68cf4d799cnxbst\" (UID: \"4e2d7c7f-f21c-41ec-aaf6-15d3976f0888\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799cnxbst" Sep 10 13:22:33.613758 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:22:33.613638 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4e2d7c7f-f21c-41ec-aaf6-15d3976f0888-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-68cf4d799cnxbst\" (UID: \"4e2d7c7f-f21c-41ec-aaf6-15d3976f0888\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799cnxbst" Sep 10 13:22:33.613758 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:22:33.613655 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4e2d7c7f-f21c-41ec-aaf6-15d3976f0888-istio-data\") pod \"data-science-gateway-data-science-gateway-class-68cf4d799cnxbst\" (UID: \"4e2d7c7f-f21c-41ec-aaf6-15d3976f0888\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799cnxbst" Sep 10 13:22:33.613758 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:22:33.613674 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4e2d7c7f-f21c-41ec-aaf6-15d3976f0888-istio-token\") pod \"data-science-gateway-data-science-gateway-class-68cf4d799cnxbst\" (UID: \"4e2d7c7f-f21c-41ec-aaf6-15d3976f0888\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799cnxbst" Sep 10 13:22:33.613758 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:22:33.613722 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4e2d7c7f-f21c-41ec-aaf6-15d3976f0888-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-68cf4d799cnxbst\" (UID: \"4e2d7c7f-f21c-41ec-aaf6-15d3976f0888\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799cnxbst" Sep 10 13:22:33.613908 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:22:33.613761 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4e2d7c7f-f21c-41ec-aaf6-15d3976f0888-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-68cf4d799cnxbst\" (UID: \"4e2d7c7f-f21c-41ec-aaf6-15d3976f0888\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799cnxbst" Sep 10 13:22:33.621514 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:22:33.621492 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-fddd6c88c-gf8ms" Sep 10 13:22:33.715016 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:22:33.714982 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4e2d7c7f-f21c-41ec-aaf6-15d3976f0888-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-68cf4d799cnxbst\" (UID: \"4e2d7c7f-f21c-41ec-aaf6-15d3976f0888\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799cnxbst" Sep 10 13:22:33.715016 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:22:33.715018 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4e2d7c7f-f21c-41ec-aaf6-15d3976f0888-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-68cf4d799cnxbst\" (UID: \"4e2d7c7f-f21c-41ec-aaf6-15d3976f0888\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799cnxbst" Sep 10 13:22:33.715218 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:22:33.715035 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4e2d7c7f-f21c-41ec-aaf6-15d3976f0888-istio-data\") pod \"data-science-gateway-data-science-gateway-class-68cf4d799cnxbst\" (UID: \"4e2d7c7f-f21c-41ec-aaf6-15d3976f0888\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799cnxbst" Sep 10 13:22:33.715218 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:22:33.715160 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4e2d7c7f-f21c-41ec-aaf6-15d3976f0888-istio-token\") pod \"data-science-gateway-data-science-gateway-class-68cf4d799cnxbst\" (UID: \"4e2d7c7f-f21c-41ec-aaf6-15d3976f0888\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799cnxbst" Sep 10 13:22:33.715322 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:22:33.715223 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4e2d7c7f-f21c-41ec-aaf6-15d3976f0888-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-68cf4d799cnxbst\" (UID: \"4e2d7c7f-f21c-41ec-aaf6-15d3976f0888\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799cnxbst" Sep 10 13:22:33.715322 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:22:33.715269 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4e2d7c7f-f21c-41ec-aaf6-15d3976f0888-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-68cf4d799cnxbst\" (UID: \"4e2d7c7f-f21c-41ec-aaf6-15d3976f0888\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799cnxbst" Sep 10 13:22:33.715461 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:22:33.715330 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4e2d7c7f-f21c-41ec-aaf6-15d3976f0888-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-68cf4d799cnxbst\" (UID: \"4e2d7c7f-f21c-41ec-aaf6-15d3976f0888\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799cnxbst" Sep 10 13:22:33.715461 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:22:33.715368 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4e2d7c7f-f21c-41ec-aaf6-15d3976f0888-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-68cf4d799cnxbst\" (UID: \"4e2d7c7f-f21c-41ec-aaf6-15d3976f0888\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799cnxbst" Sep 10 13:22:33.715461 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:22:33.715378 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4e2d7c7f-f21c-41ec-aaf6-15d3976f0888-istio-data\") pod \"data-science-gateway-data-science-gateway-class-68cf4d799cnxbst\" (UID: \"4e2d7c7f-f21c-41ec-aaf6-15d3976f0888\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799cnxbst" Sep 10 13:22:33.715461 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:22:33.715399 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-47l76\" (UniqueName: \"kubernetes.io/projected/4e2d7c7f-f21c-41ec-aaf6-15d3976f0888-kube-api-access-47l76\") pod \"data-science-gateway-data-science-gateway-class-68cf4d799cnxbst\" (UID: \"4e2d7c7f-f21c-41ec-aaf6-15d3976f0888\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799cnxbst" Sep 10 13:22:33.715461 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:22:33.715446 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4e2d7c7f-f21c-41ec-aaf6-15d3976f0888-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-68cf4d799cnxbst\" (UID: \"4e2d7c7f-f21c-41ec-aaf6-15d3976f0888\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799cnxbst" Sep 10 13:22:33.715700 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:22:33.715491 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4e2d7c7f-f21c-41ec-aaf6-15d3976f0888-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-68cf4d799cnxbst\" (UID: \"4e2d7c7f-f21c-41ec-aaf6-15d3976f0888\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799cnxbst" Sep 10 13:22:33.715700 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:22:33.715636 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4e2d7c7f-f21c-41ec-aaf6-15d3976f0888-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-68cf4d799cnxbst\" (UID: \"4e2d7c7f-f21c-41ec-aaf6-15d3976f0888\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799cnxbst" Sep 10 13:22:33.716018 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:22:33.715999 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4e2d7c7f-f21c-41ec-aaf6-15d3976f0888-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-68cf4d799cnxbst\" (UID: \"4e2d7c7f-f21c-41ec-aaf6-15d3976f0888\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799cnxbst" Sep 10 13:22:33.717624 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:22:33.717603 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4e2d7c7f-f21c-41ec-aaf6-15d3976f0888-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-68cf4d799cnxbst\" (UID: \"4e2d7c7f-f21c-41ec-aaf6-15d3976f0888\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799cnxbst" Sep 10 13:22:33.717782 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:22:33.717764 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4e2d7c7f-f21c-41ec-aaf6-15d3976f0888-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-68cf4d799cnxbst\" (UID: \"4e2d7c7f-f21c-41ec-aaf6-15d3976f0888\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799cnxbst" Sep 10 13:22:33.727231 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:22:33.727199 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4e2d7c7f-f21c-41ec-aaf6-15d3976f0888-istio-token\") pod \"data-science-gateway-data-science-gateway-class-68cf4d799cnxbst\" (UID: \"4e2d7c7f-f21c-41ec-aaf6-15d3976f0888\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799cnxbst" Sep 10 13:22:33.727405 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:22:33.727387 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-47l76\" (UniqueName: \"kubernetes.io/projected/4e2d7c7f-f21c-41ec-aaf6-15d3976f0888-kube-api-access-47l76\") pod \"data-science-gateway-data-science-gateway-class-68cf4d799cnxbst\" (UID: \"4e2d7c7f-f21c-41ec-aaf6-15d3976f0888\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799cnxbst" Sep 10 13:22:33.810522 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:22:33.810497 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799cnxbst" Sep 10 13:22:33.984615 ip-10-0-139-174 kubenswrapper[2557]: W0910 13:22:33.984585 2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod30d85e7c_08a7_41a2_beb0_4a5b33c2f826.slice/crio-9423920e60fd48baaed7d5d38a8dfa88c3307886df989701e2c43aab32cb2e9f WatchSource:0}: Error finding container 9423920e60fd48baaed7d5d38a8dfa88c3307886df989701e2c43aab32cb2e9f: Status 404 returned error can't find the container with id 9423920e60fd48baaed7d5d38a8dfa88c3307886df989701e2c43aab32cb2e9f Sep 10 13:22:33.986951 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:22:33.986931 2557 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Sep 10 13:22:33.991895 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:22:33.991872 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-fddd6c88c-gf8ms"] Sep 10 13:22:34.197360 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:22:34.197323 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799cnxbst"] Sep 10 13:22:34.201533 ip-10-0-139-174 kubenswrapper[2557]: W0910 13:22:34.201506 2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4e2d7c7f_f21c_41ec_aaf6_15d3976f0888.slice/crio-e0755740f51d6f48b20930b17631b56fabde04991a2459eaa498eb5f4d646efb WatchSource:0}: Error finding container e0755740f51d6f48b20930b17631b56fabde04991a2459eaa498eb5f4d646efb: Status 404 returned error can't find the container with id e0755740f51d6f48b20930b17631b56fabde04991a2459eaa498eb5f4d646efb Sep 10 13:22:34.333455 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:22:34.333357 2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/odh-model-controller-7998c67dc4-9rt5c"] Sep 10 13:22:34.339485 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:22:34.338149 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/odh-model-controller-7998c67dc4-9rt5c" Sep 10 13:22:34.341847 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:22:34.341807 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"odh-model-controller-webhook-cert\"" Sep 10 13:22:34.341973 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:22:34.341940 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"odh-model-controller-dockercfg-7xg7d\"" Sep 10 13:22:34.365327 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:22:34.365301 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/odh-model-controller-7998c67dc4-9rt5c"] Sep 10 13:22:34.419585 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:22:34.419558 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/f7115eea-d966-4e45-b0d4-ab798ec09ef2-cert\") pod \"odh-model-controller-7998c67dc4-9rt5c\" (UID: \"f7115eea-d966-4e45-b0d4-ab798ec09ef2\") " pod="opendatahub/odh-model-controller-7998c67dc4-9rt5c" Sep 10 13:22:34.419707 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:22:34.419590 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-snvn5\" (UniqueName: \"kubernetes.io/projected/f7115eea-d966-4e45-b0d4-ab798ec09ef2-kube-api-access-snvn5\") pod \"odh-model-controller-7998c67dc4-9rt5c\" (UID: \"f7115eea-d966-4e45-b0d4-ab798ec09ef2\") " pod="opendatahub/odh-model-controller-7998c67dc4-9rt5c" Sep 10 13:22:34.520694 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:22:34.520667 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/f7115eea-d966-4e45-b0d4-ab798ec09ef2-cert\") pod \"odh-model-controller-7998c67dc4-9rt5c\" (UID: \"f7115eea-d966-4e45-b0d4-ab798ec09ef2\") " pod="opendatahub/odh-model-controller-7998c67dc4-9rt5c" Sep 10 13:22:34.520694 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:22:34.520701 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-snvn5\" (UniqueName: \"kubernetes.io/projected/f7115eea-d966-4e45-b0d4-ab798ec09ef2-kube-api-access-snvn5\") pod \"odh-model-controller-7998c67dc4-9rt5c\" (UID: \"f7115eea-d966-4e45-b0d4-ab798ec09ef2\") " pod="opendatahub/odh-model-controller-7998c67dc4-9rt5c" Sep 10 13:22:34.520884 ip-10-0-139-174 kubenswrapper[2557]: E0910 13:22:34.520825 2557 secret.go:189] Couldn't get secret opendatahub/odh-model-controller-webhook-cert: secret "odh-model-controller-webhook-cert" not found Sep 10 13:22:34.520884 ip-10-0-139-174 kubenswrapper[2557]: E0910 13:22:34.520881 2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f7115eea-d966-4e45-b0d4-ab798ec09ef2-cert podName:f7115eea-d966-4e45-b0d4-ab798ec09ef2 nodeName:}" failed. No retries permitted until 2026-09-10 13:22:35.020865995 +0000 UTC m=+315.327860386 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/f7115eea-d966-4e45-b0d4-ab798ec09ef2-cert") pod "odh-model-controller-7998c67dc4-9rt5c" (UID: "f7115eea-d966-4e45-b0d4-ab798ec09ef2") : secret "odh-model-controller-webhook-cert" not found Sep 10 13:22:34.535295 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:22:34.535268 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-snvn5\" (UniqueName: \"kubernetes.io/projected/f7115eea-d966-4e45-b0d4-ab798ec09ef2-kube-api-access-snvn5\") pod \"odh-model-controller-7998c67dc4-9rt5c\" (UID: \"f7115eea-d966-4e45-b0d4-ab798ec09ef2\") " pod="opendatahub/odh-model-controller-7998c67dc4-9rt5c" Sep 10 13:22:34.669644 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:22:34.669543 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-fddd6c88c-gf8ms" event={"ID":"30d85e7c-08a7-41a2-beb0-4a5b33c2f826","Type":"ContainerStarted","Data":"9423920e60fd48baaed7d5d38a8dfa88c3307886df989701e2c43aab32cb2e9f"} Sep 10 13:22:34.670828 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:22:34.670799 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799cnxbst" event={"ID":"4e2d7c7f-f21c-41ec-aaf6-15d3976f0888","Type":"ContainerStarted","Data":"e0755740f51d6f48b20930b17631b56fabde04991a2459eaa498eb5f4d646efb"} Sep 10 13:22:35.024891 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:22:35.024858 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/f7115eea-d966-4e45-b0d4-ab798ec09ef2-cert\") pod \"odh-model-controller-7998c67dc4-9rt5c\" (UID: \"f7115eea-d966-4e45-b0d4-ab798ec09ef2\") " pod="opendatahub/odh-model-controller-7998c67dc4-9rt5c" Sep 10 13:22:35.027832 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:22:35.027758 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/f7115eea-d966-4e45-b0d4-ab798ec09ef2-cert\") pod \"odh-model-controller-7998c67dc4-9rt5c\" (UID: \"f7115eea-d966-4e45-b0d4-ab798ec09ef2\") " pod="opendatahub/odh-model-controller-7998c67dc4-9rt5c" Sep 10 13:22:35.249100 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:22:35.249068 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/odh-model-controller-7998c67dc4-9rt5c" Sep 10 13:22:35.287754 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:22:35.287629 2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/model-serving-api-54d89bd5fc-fc5qw"] Sep 10 13:22:35.290395 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:22:35.290366 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/model-serving-api-54d89bd5fc-fc5qw" Sep 10 13:22:35.293214 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:22:35.293189 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"model-serving-api-tls\"" Sep 10 13:22:35.310673 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:22:35.310643 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/model-serving-api-54d89bd5fc-fc5qw"] Sep 10 13:22:35.328871 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:22:35.328717 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/05ac257e-0d9e-4dfa-9309-a0d065304581-tls-certs\") pod \"model-serving-api-54d89bd5fc-fc5qw\" (UID: \"05ac257e-0d9e-4dfa-9309-a0d065304581\") " pod="opendatahub/model-serving-api-54d89bd5fc-fc5qw" Sep 10 13:22:35.328871 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:22:35.328788 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j4b5r\" (UniqueName: \"kubernetes.io/projected/05ac257e-0d9e-4dfa-9309-a0d065304581-kube-api-access-j4b5r\") pod \"model-serving-api-54d89bd5fc-fc5qw\" (UID: \"05ac257e-0d9e-4dfa-9309-a0d065304581\") " pod="opendatahub/model-serving-api-54d89bd5fc-fc5qw" Sep 10 13:22:35.430330 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:22:35.430294 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j4b5r\" (UniqueName: \"kubernetes.io/projected/05ac257e-0d9e-4dfa-9309-a0d065304581-kube-api-access-j4b5r\") pod \"model-serving-api-54d89bd5fc-fc5qw\" (UID: \"05ac257e-0d9e-4dfa-9309-a0d065304581\") " pod="opendatahub/model-serving-api-54d89bd5fc-fc5qw" Sep 10 13:22:35.430822 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:22:35.430434 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/05ac257e-0d9e-4dfa-9309-a0d065304581-tls-certs\") pod \"model-serving-api-54d89bd5fc-fc5qw\" (UID: \"05ac257e-0d9e-4dfa-9309-a0d065304581\") " pod="opendatahub/model-serving-api-54d89bd5fc-fc5qw" Sep 10 13:22:35.430822 ip-10-0-139-174 kubenswrapper[2557]: E0910 13:22:35.430585 2557 secret.go:189] Couldn't get secret opendatahub/model-serving-api-tls: secret "model-serving-api-tls" not found Sep 10 13:22:35.430822 ip-10-0-139-174 kubenswrapper[2557]: E0910 13:22:35.430658 2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/05ac257e-0d9e-4dfa-9309-a0d065304581-tls-certs podName:05ac257e-0d9e-4dfa-9309-a0d065304581 nodeName:}" failed. No retries permitted until 2026-09-10 13:22:35.930635716 +0000 UTC m=+316.237630128 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "tls-certs" (UniqueName: "kubernetes.io/secret/05ac257e-0d9e-4dfa-9309-a0d065304581-tls-certs") pod "model-serving-api-54d89bd5fc-fc5qw" (UID: "05ac257e-0d9e-4dfa-9309-a0d065304581") : secret "model-serving-api-tls" not found Sep 10 13:22:35.443836 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:22:35.443792 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-j4b5r\" (UniqueName: \"kubernetes.io/projected/05ac257e-0d9e-4dfa-9309-a0d065304581-kube-api-access-j4b5r\") pod \"model-serving-api-54d89bd5fc-fc5qw\" (UID: \"05ac257e-0d9e-4dfa-9309-a0d065304581\") " pod="opendatahub/model-serving-api-54d89bd5fc-fc5qw" Sep 10 13:22:35.555152 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:22:35.555112 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/odh-model-controller-7998c67dc4-9rt5c"] Sep 10 13:22:35.557465 ip-10-0-139-174 kubenswrapper[2557]: W0910 13:22:35.557400 2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf7115eea_d966_4e45_b0d4_ab798ec09ef2.slice/crio-5ef0aa3bf498466ba3a9037eb03ad41474b4bb43d00df042263fbd357ecf36aa WatchSource:0}: Error finding container 5ef0aa3bf498466ba3a9037eb03ad41474b4bb43d00df042263fbd357ecf36aa: Status 404 returned error can't find the container with id 5ef0aa3bf498466ba3a9037eb03ad41474b4bb43d00df042263fbd357ecf36aa Sep 10 13:22:35.678778 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:22:35.678730 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/odh-model-controller-7998c67dc4-9rt5c" event={"ID":"f7115eea-d966-4e45-b0d4-ab798ec09ef2","Type":"ContainerStarted","Data":"5ef0aa3bf498466ba3a9037eb03ad41474b4bb43d00df042263fbd357ecf36aa"} Sep 10 13:22:35.935695 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:22:35.935618 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/05ac257e-0d9e-4dfa-9309-a0d065304581-tls-certs\") pod \"model-serving-api-54d89bd5fc-fc5qw\" (UID: \"05ac257e-0d9e-4dfa-9309-a0d065304581\") " pod="opendatahub/model-serving-api-54d89bd5fc-fc5qw" Sep 10 13:22:35.938269 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:22:35.938244 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/05ac257e-0d9e-4dfa-9309-a0d065304581-tls-certs\") pod \"model-serving-api-54d89bd5fc-fc5qw\" (UID: \"05ac257e-0d9e-4dfa-9309-a0d065304581\") " pod="opendatahub/model-serving-api-54d89bd5fc-fc5qw" Sep 10 13:22:36.205377 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:22:36.205341 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/model-serving-api-54d89bd5fc-fc5qw" Sep 10 13:22:37.383082 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:22:37.383061 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/model-serving-api-54d89bd5fc-fc5qw"] Sep 10 13:22:37.385075 ip-10-0-139-174 kubenswrapper[2557]: W0910 13:22:37.385049 2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod05ac257e_0d9e_4dfa_9309_a0d065304581.slice/crio-4a85145c413165de4c1fac2a1a8c8f258eb40e76156689e2e03d35b487cd49e3 WatchSource:0}: Error finding container 4a85145c413165de4c1fac2a1a8c8f258eb40e76156689e2e03d35b487cd49e3: Status 404 returned error can't find the container with id 4a85145c413165de4c1fac2a1a8c8f258eb40e76156689e2e03d35b487cd49e3 Sep 10 13:22:37.409680 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:22:37.409640 2557 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892152Ki","pods":"250"} Sep 10 13:22:37.409794 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:22:37.409720 2557 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892152Ki","pods":"250"} Sep 10 13:22:37.409794 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:22:37.409752 2557 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892152Ki","pods":"250"} Sep 10 13:22:37.415311 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:22:37.415284 2557 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892152Ki","pods":"250"} Sep 10 13:22:37.415373 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:22:37.415343 2557 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892152Ki","pods":"250"} Sep 10 13:22:37.415409 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:22:37.415388 2557 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892152Ki","pods":"250"} Sep 10 13:22:37.692605 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:22:37.692487 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/model-serving-api-54d89bd5fc-fc5qw" event={"ID":"05ac257e-0d9e-4dfa-9309-a0d065304581","Type":"ContainerStarted","Data":"4a85145c413165de4c1fac2a1a8c8f258eb40e76156689e2e03d35b487cd49e3"} Sep 10 13:22:37.695235 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:22:37.695176 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799cnxbst" event={"ID":"4e2d7c7f-f21c-41ec-aaf6-15d3976f0888","Type":"ContainerStarted","Data":"33a07bbdf809cef9b53f8755cc0ca89b55f783ef55b03672a739e65e8e55431a"} Sep 10 13:22:37.697561 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:22:37.697496 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-fddd6c88c-gf8ms" event={"ID":"30d85e7c-08a7-41a2-beb0-4a5b33c2f826","Type":"ContainerStarted","Data":"08d35bd338141e95053ece69a357e2178eb5f14f2de92dce47691db334eac786"} Sep 10 13:22:37.734445 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:22:37.734351 2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799cnxbst" podStartSLOduration=1.528606253 podStartE2EDuration="4.734335797s" totalImagesPullingTime="3.205729544s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 13:22:33 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 13:22:37.732528111 +0000 UTC m=+318.039522526" watchObservedRunningTime="2026-09-10 13:22:37.734335797 +0000 UTC m=+318.041330208" Sep 10 13:22:37.784931 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:22:37.784699 2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-fddd6c88c-gf8ms" podStartSLOduration=1.356640895 podStartE2EDuration="4.784678499s" totalImagesPullingTime="3.428037604s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 13:22:33 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 13:22:37.783649804 +0000 UTC m=+318.090644219" watchObservedRunningTime="2026-09-10 13:22:37.784678499 +0000 UTC m=+318.091672916" Sep 10 13:22:37.810692 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:22:37.810610 2557 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799cnxbst" Sep 10 13:22:38.622661 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:22:38.622581 2557 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-fddd6c88c-gf8ms" Sep 10 13:22:38.624365 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:22:38.624232 2557 patch_prober.go:28] interesting pod/data-science-gateway-data-science-gateway-class-fddd6c88c-gf8ms container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.28:15021/healthz/ready\": dial tcp 10.134.0.28:15021: connect: connection refused" start-of-body= Sep 10 13:22:38.624365 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:22:38.624302 2557 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-fddd6c88c-gf8ms" podUID="30d85e7c-08a7-41a2-beb0-4a5b33c2f826" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.28:15021/healthz/ready\": dial tcp 10.134.0.28:15021: connect: connection refused" Sep 10 13:22:38.816763 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:22:38.816727 2557 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799cnxbst" Sep 10 13:22:39.622554 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:22:39.622522 2557 patch_prober.go:28] interesting pod/data-science-gateway-data-science-gateway-class-fddd6c88c-gf8ms container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.28:15021/healthz/ready\": dial tcp 10.134.0.28:15021: connect: connection refused" start-of-body= Sep 10 13:22:39.622731 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:22:39.622587 2557 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-fddd6c88c-gf8ms" podUID="30d85e7c-08a7-41a2-beb0-4a5b33c2f826" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.28:15021/healthz/ready\": dial tcp 10.134.0.28:15021: connect: connection refused" Sep 10 13:22:39.709721 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:22:39.709685 2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799cnxbst" Sep 10 13:22:39.710770 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:22:39.710747 2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799cnxbst" Sep 10 13:22:39.781123 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:22:39.781089 2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-fddd6c88c-gf8ms"] Sep 10 13:22:40.621924 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:22:40.621887 2557 patch_prober.go:28] interesting pod/data-science-gateway-data-science-gateway-class-fddd6c88c-gf8ms container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.28:15021/healthz/ready\": dial tcp 10.134.0.28:15021: connect: connection refused" start-of-body= Sep 10 13:22:40.622100 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:22:40.621959 2557 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-fddd6c88c-gf8ms" podUID="30d85e7c-08a7-41a2-beb0-4a5b33c2f826" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.28:15021/healthz/ready\": dial tcp 10.134.0.28:15021: connect: connection refused" Sep 10 13:22:40.716057 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:22:40.716019 2557 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-fddd6c88c-gf8ms" podUID="30d85e7c-08a7-41a2-beb0-4a5b33c2f826" containerName="istio-proxy" containerID="cri-o://08d35bd338141e95053ece69a357e2178eb5f14f2de92dce47691db334eac786" gracePeriod=30 Sep 10 13:22:41.727504 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:22:41.727468 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/model-serving-api-54d89bd5fc-fc5qw" event={"ID":"05ac257e-0d9e-4dfa-9309-a0d065304581","Type":"ContainerStarted","Data":"6ca6f9edc8eb2f5f12c1bc5bea87f3734e30d788b9f323b682daf115ecc20be7"} Sep 10 13:22:41.727993 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:22:41.727559 2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/model-serving-api-54d89bd5fc-fc5qw" Sep 10 13:22:41.729069 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:22:41.729038 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/odh-model-controller-7998c67dc4-9rt5c" event={"ID":"f7115eea-d966-4e45-b0d4-ab798ec09ef2","Type":"ContainerStarted","Data":"d8b88ab75f77d70e1d769c54472bb4866e80da132261d3baa7672ab757948227"} Sep 10 13:22:41.729293 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:22:41.729277 2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/odh-model-controller-7998c67dc4-9rt5c" Sep 10 13:22:41.803519 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:22:41.803468 2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/model-serving-api-54d89bd5fc-fc5qw" podStartSLOduration=3.12986581 podStartE2EDuration="6.803452075s" totalImagesPullingTime="3.673586265s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 13:22:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 13:22:41.800854669 +0000 UTC m=+322.107849082" watchObservedRunningTime="2026-09-10 13:22:41.803452075 +0000 UTC m=+322.110446505" Sep 10 13:22:41.857220 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:22:41.857170 2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/odh-model-controller-7998c67dc4-9rt5c" podStartSLOduration=2.318944485 podStartE2EDuration="7.857150754s" totalImagesPullingTime="5.538206269s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 13:22:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 13:22:41.856162521 +0000 UTC m=+322.163156936" watchObservedRunningTime="2026-09-10 13:22:41.857150754 +0000 UTC m=+322.164145169" Sep 10 13:22:43.390127 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:22:43.390092 2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-5677ccb49b-tkt56"] Sep 10 13:22:43.392461 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:22:43.392443 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-tkt56" Sep 10 13:22:43.396630 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:22:43.396611 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-openshift-gateway-dockercfg-9466g\"" Sep 10 13:22:43.398651 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:22:43.398631 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"cacerts\"" Sep 10 13:22:43.398768 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:22:43.398638 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istio-kubeconfig\"" Sep 10 13:22:43.399329 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:22:43.399308 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-tls\"" Sep 10 13:22:43.425083 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:22:43.425051 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-5677ccb49b-tkt56"] Sep 10 13:22:43.504564 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:22:43.504531 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/d978ed31-3a9b-4efb-b5c9-da53484bebc9-istio-kubeconfig\") pod \"istiod-openshift-gateway-5677ccb49b-tkt56\" (UID: \"d978ed31-3a9b-4efb-b5c9-da53484bebc9\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-tkt56" Sep 10 13:22:43.504709 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:22:43.504583 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/d978ed31-3a9b-4efb-b5c9-da53484bebc9-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-5677ccb49b-tkt56\" (UID: \"d978ed31-3a9b-4efb-b5c9-da53484bebc9\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-tkt56" Sep 10 13:22:43.504709 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:22:43.504653 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/d978ed31-3a9b-4efb-b5c9-da53484bebc9-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-5677ccb49b-tkt56\" (UID: \"d978ed31-3a9b-4efb-b5c9-da53484bebc9\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-tkt56" Sep 10 13:22:43.504812 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:22:43.504734 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/d978ed31-3a9b-4efb-b5c9-da53484bebc9-local-certs\") pod \"istiod-openshift-gateway-5677ccb49b-tkt56\" (UID: \"d978ed31-3a9b-4efb-b5c9-da53484bebc9\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-tkt56" Sep 10 13:22:43.504812 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:22:43.504768 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/d978ed31-3a9b-4efb-b5c9-da53484bebc9-cacerts\") pod \"istiod-openshift-gateway-5677ccb49b-tkt56\" (UID: \"d978ed31-3a9b-4efb-b5c9-da53484bebc9\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-tkt56" Sep 10 13:22:43.504889 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:22:43.504818 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c6bgb\" (UniqueName: \"kubernetes.io/projected/d978ed31-3a9b-4efb-b5c9-da53484bebc9-kube-api-access-c6bgb\") pod \"istiod-openshift-gateway-5677ccb49b-tkt56\" (UID: \"d978ed31-3a9b-4efb-b5c9-da53484bebc9\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-tkt56" Sep 10 13:22:43.504889 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:22:43.504837 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d978ed31-3a9b-4efb-b5c9-da53484bebc9-istio-token\") pod \"istiod-openshift-gateway-5677ccb49b-tkt56\" (UID: \"d978ed31-3a9b-4efb-b5c9-da53484bebc9\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-tkt56" Sep 10 13:22:43.605692 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:22:43.605645 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/d978ed31-3a9b-4efb-b5c9-da53484bebc9-local-certs\") pod \"istiod-openshift-gateway-5677ccb49b-tkt56\" (UID: \"d978ed31-3a9b-4efb-b5c9-da53484bebc9\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-tkt56" Sep 10 13:22:43.605877 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:22:43.605700 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/d978ed31-3a9b-4efb-b5c9-da53484bebc9-cacerts\") pod \"istiod-openshift-gateway-5677ccb49b-tkt56\" (UID: \"d978ed31-3a9b-4efb-b5c9-da53484bebc9\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-tkt56" Sep 10 13:22:43.605877 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:22:43.605752 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c6bgb\" (UniqueName: \"kubernetes.io/projected/d978ed31-3a9b-4efb-b5c9-da53484bebc9-kube-api-access-c6bgb\") pod \"istiod-openshift-gateway-5677ccb49b-tkt56\" (UID: \"d978ed31-3a9b-4efb-b5c9-da53484bebc9\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-tkt56" Sep 10 13:22:43.605877 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:22:43.605782 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d978ed31-3a9b-4efb-b5c9-da53484bebc9-istio-token\") pod \"istiod-openshift-gateway-5677ccb49b-tkt56\" (UID: \"d978ed31-3a9b-4efb-b5c9-da53484bebc9\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-tkt56" Sep 10 13:22:43.605877 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:22:43.605821 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/d978ed31-3a9b-4efb-b5c9-da53484bebc9-istio-kubeconfig\") pod \"istiod-openshift-gateway-5677ccb49b-tkt56\" (UID: \"d978ed31-3a9b-4efb-b5c9-da53484bebc9\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-tkt56" Sep 10 13:22:43.605877 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:22:43.605856 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/d978ed31-3a9b-4efb-b5c9-da53484bebc9-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-5677ccb49b-tkt56\" (UID: \"d978ed31-3a9b-4efb-b5c9-da53484bebc9\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-tkt56" Sep 10 13:22:43.606150 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:22:43.605908 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/d978ed31-3a9b-4efb-b5c9-da53484bebc9-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-5677ccb49b-tkt56\" (UID: \"d978ed31-3a9b-4efb-b5c9-da53484bebc9\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-tkt56" Sep 10 13:22:43.606925 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:22:43.606893 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/d978ed31-3a9b-4efb-b5c9-da53484bebc9-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-5677ccb49b-tkt56\" (UID: \"d978ed31-3a9b-4efb-b5c9-da53484bebc9\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-tkt56" Sep 10 13:22:43.608318 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:22:43.608291 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/d978ed31-3a9b-4efb-b5c9-da53484bebc9-cacerts\") pod \"istiod-openshift-gateway-5677ccb49b-tkt56\" (UID: \"d978ed31-3a9b-4efb-b5c9-da53484bebc9\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-tkt56" Sep 10 13:22:43.608518 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:22:43.608496 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/d978ed31-3a9b-4efb-b5c9-da53484bebc9-istio-kubeconfig\") pod \"istiod-openshift-gateway-5677ccb49b-tkt56\" (UID: \"d978ed31-3a9b-4efb-b5c9-da53484bebc9\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-tkt56" Sep 10 13:22:43.608617 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:22:43.608594 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/d978ed31-3a9b-4efb-b5c9-da53484bebc9-local-certs\") pod \"istiod-openshift-gateway-5677ccb49b-tkt56\" (UID: \"d978ed31-3a9b-4efb-b5c9-da53484bebc9\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-tkt56" Sep 10 13:22:43.608729 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:22:43.608714 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/d978ed31-3a9b-4efb-b5c9-da53484bebc9-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-5677ccb49b-tkt56\" (UID: \"d978ed31-3a9b-4efb-b5c9-da53484bebc9\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-tkt56" Sep 10 13:22:43.621218 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:22:43.621189 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-c6bgb\" (UniqueName: \"kubernetes.io/projected/d978ed31-3a9b-4efb-b5c9-da53484bebc9-kube-api-access-c6bgb\") pod \"istiod-openshift-gateway-5677ccb49b-tkt56\" (UID: \"d978ed31-3a9b-4efb-b5c9-da53484bebc9\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-tkt56" Sep 10 13:22:43.624263 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:22:43.624241 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d978ed31-3a9b-4efb-b5c9-da53484bebc9-istio-token\") pod \"istiod-openshift-gateway-5677ccb49b-tkt56\" (UID: \"d978ed31-3a9b-4efb-b5c9-da53484bebc9\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-tkt56" Sep 10 13:22:43.701466 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:22:43.701447 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-tkt56" Sep 10 13:22:44.006765 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:22:44.006740 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-5677ccb49b-tkt56"] Sep 10 13:22:44.008751 ip-10-0-139-174 kubenswrapper[2557]: W0910 13:22:44.008718 2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd978ed31_3a9b_4efb_b5c9_da53484bebc9.slice/crio-7bd4360f068407f47e6633e4038dfcc451cdeb463eef75115469edae4408b2fe WatchSource:0}: Error finding container 7bd4360f068407f47e6633e4038dfcc451cdeb463eef75115469edae4408b2fe: Status 404 returned error can't find the container with id 7bd4360f068407f47e6633e4038dfcc451cdeb463eef75115469edae4408b2fe Sep 10 13:22:44.748388 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:22:44.748354 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-tkt56" event={"ID":"d978ed31-3a9b-4efb-b5c9-da53484bebc9","Type":"ContainerStarted","Data":"7bd4360f068407f47e6633e4038dfcc451cdeb463eef75115469edae4408b2fe"} Sep 10 13:22:46.199278 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:22:46.199222 2557 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892152Ki","pods":"250"} Sep 10 13:22:46.199682 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:22:46.199297 2557 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892152Ki","pods":"250"} Sep 10 13:22:46.309950 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:22:46.309926 2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-fddd6c88c-gf8ms" Sep 10 13:22:46.432559 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:22:46.432474 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/30d85e7c-08a7-41a2-beb0-4a5b33c2f826-istio-podinfo\") pod \"30d85e7c-08a7-41a2-beb0-4a5b33c2f826\" (UID: \"30d85e7c-08a7-41a2-beb0-4a5b33c2f826\") " Sep 10 13:22:46.432559 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:22:46.432523 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hv5dl\" (UniqueName: \"kubernetes.io/projected/30d85e7c-08a7-41a2-beb0-4a5b33c2f826-kube-api-access-hv5dl\") pod \"30d85e7c-08a7-41a2-beb0-4a5b33c2f826\" (UID: \"30d85e7c-08a7-41a2-beb0-4a5b33c2f826\") " Sep 10 13:22:46.432559 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:22:46.432544 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/30d85e7c-08a7-41a2-beb0-4a5b33c2f826-workload-certs\") pod \"30d85e7c-08a7-41a2-beb0-4a5b33c2f826\" (UID: \"30d85e7c-08a7-41a2-beb0-4a5b33c2f826\") " Sep 10 13:22:46.432869 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:22:46.432564 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/30d85e7c-08a7-41a2-beb0-4a5b33c2f826-istio-token\") pod \"30d85e7c-08a7-41a2-beb0-4a5b33c2f826\" (UID: \"30d85e7c-08a7-41a2-beb0-4a5b33c2f826\") " Sep 10 13:22:46.432869 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:22:46.432630 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/30d85e7c-08a7-41a2-beb0-4a5b33c2f826-istiod-ca-cert\") pod \"30d85e7c-08a7-41a2-beb0-4a5b33c2f826\" (UID: \"30d85e7c-08a7-41a2-beb0-4a5b33c2f826\") " Sep 10 13:22:46.432869 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:22:46.432662 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/30d85e7c-08a7-41a2-beb0-4a5b33c2f826-istio-envoy\") pod \"30d85e7c-08a7-41a2-beb0-4a5b33c2f826\" (UID: \"30d85e7c-08a7-41a2-beb0-4a5b33c2f826\") " Sep 10 13:22:46.432869 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:22:46.432697 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/30d85e7c-08a7-41a2-beb0-4a5b33c2f826-workload-socket\") pod \"30d85e7c-08a7-41a2-beb0-4a5b33c2f826\" (UID: \"30d85e7c-08a7-41a2-beb0-4a5b33c2f826\") " Sep 10 13:22:46.432869 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:22:46.432824 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/30d85e7c-08a7-41a2-beb0-4a5b33c2f826-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "30d85e7c-08a7-41a2-beb0-4a5b33c2f826" (UID: "30d85e7c-08a7-41a2-beb0-4a5b33c2f826"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 13:22:46.432869 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:22:46.432859 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/30d85e7c-08a7-41a2-beb0-4a5b33c2f826-istio-data\") pod \"30d85e7c-08a7-41a2-beb0-4a5b33c2f826\" (UID: \"30d85e7c-08a7-41a2-beb0-4a5b33c2f826\") " Sep 10 13:22:46.433159 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:22:46.432909 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/30d85e7c-08a7-41a2-beb0-4a5b33c2f826-credential-socket\") pod \"30d85e7c-08a7-41a2-beb0-4a5b33c2f826\" (UID: \"30d85e7c-08a7-41a2-beb0-4a5b33c2f826\") " Sep 10 13:22:46.433159 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:22:46.433056 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/30d85e7c-08a7-41a2-beb0-4a5b33c2f826-istio-data" (OuterVolumeSpecName: "istio-data") pod "30d85e7c-08a7-41a2-beb0-4a5b33c2f826" (UID: "30d85e7c-08a7-41a2-beb0-4a5b33c2f826"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 13:22:46.433159 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:22:46.433072 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/30d85e7c-08a7-41a2-beb0-4a5b33c2f826-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "30d85e7c-08a7-41a2-beb0-4a5b33c2f826" (UID: "30d85e7c-08a7-41a2-beb0-4a5b33c2f826"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 13:22:46.433314 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:22:46.433228 2557 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/30d85e7c-08a7-41a2-beb0-4a5b33c2f826-workload-certs\") on node \"ip-10-0-139-174.ec2.internal\" DevicePath \"\"" Sep 10 13:22:46.433314 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:22:46.433247 2557 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/30d85e7c-08a7-41a2-beb0-4a5b33c2f826-workload-socket\") on node \"ip-10-0-139-174.ec2.internal\" DevicePath \"\"" Sep 10 13:22:46.433314 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:22:46.433248 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/30d85e7c-08a7-41a2-beb0-4a5b33c2f826-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "30d85e7c-08a7-41a2-beb0-4a5b33c2f826" (UID: "30d85e7c-08a7-41a2-beb0-4a5b33c2f826"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 13:22:46.433314 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:22:46.433261 2557 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/30d85e7c-08a7-41a2-beb0-4a5b33c2f826-istio-data\") on node \"ip-10-0-139-174.ec2.internal\" DevicePath \"\"" Sep 10 13:22:46.433561 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:22:46.433535 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/30d85e7c-08a7-41a2-beb0-4a5b33c2f826-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "30d85e7c-08a7-41a2-beb0-4a5b33c2f826" (UID: "30d85e7c-08a7-41a2-beb0-4a5b33c2f826"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 10 13:22:46.435165 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:22:46.435136 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/30d85e7c-08a7-41a2-beb0-4a5b33c2f826-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "30d85e7c-08a7-41a2-beb0-4a5b33c2f826" (UID: "30d85e7c-08a7-41a2-beb0-4a5b33c2f826"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 13:22:46.435165 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:22:46.435149 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/30d85e7c-08a7-41a2-beb0-4a5b33c2f826-istio-token" (OuterVolumeSpecName: "istio-token") pod "30d85e7c-08a7-41a2-beb0-4a5b33c2f826" (UID: "30d85e7c-08a7-41a2-beb0-4a5b33c2f826"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 10 13:22:46.435165 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:22:46.435164 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/30d85e7c-08a7-41a2-beb0-4a5b33c2f826-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "30d85e7c-08a7-41a2-beb0-4a5b33c2f826" (UID: "30d85e7c-08a7-41a2-beb0-4a5b33c2f826"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Sep 10 13:22:46.435409 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:22:46.435394 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/30d85e7c-08a7-41a2-beb0-4a5b33c2f826-kube-api-access-hv5dl" (OuterVolumeSpecName: "kube-api-access-hv5dl") pod "30d85e7c-08a7-41a2-beb0-4a5b33c2f826" (UID: "30d85e7c-08a7-41a2-beb0-4a5b33c2f826"). InnerVolumeSpecName "kube-api-access-hv5dl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 10 13:22:46.534472 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:22:46.534439 2557 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/30d85e7c-08a7-41a2-beb0-4a5b33c2f826-istio-podinfo\") on node \"ip-10-0-139-174.ec2.internal\" DevicePath \"\"" Sep 10 13:22:46.534472 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:22:46.534467 2557 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hv5dl\" (UniqueName: \"kubernetes.io/projected/30d85e7c-08a7-41a2-beb0-4a5b33c2f826-kube-api-access-hv5dl\") on node \"ip-10-0-139-174.ec2.internal\" DevicePath \"\"" Sep 10 13:22:46.534472 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:22:46.534477 2557 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/30d85e7c-08a7-41a2-beb0-4a5b33c2f826-istio-token\") on node \"ip-10-0-139-174.ec2.internal\" DevicePath \"\"" Sep 10 13:22:46.534670 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:22:46.534486 2557 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/30d85e7c-08a7-41a2-beb0-4a5b33c2f826-istiod-ca-cert\") on node \"ip-10-0-139-174.ec2.internal\" DevicePath \"\"" Sep 10 13:22:46.534670 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:22:46.534494 2557 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/30d85e7c-08a7-41a2-beb0-4a5b33c2f826-istio-envoy\") on node \"ip-10-0-139-174.ec2.internal\" DevicePath \"\"" Sep 10 13:22:46.534670 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:22:46.534502 2557 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/30d85e7c-08a7-41a2-beb0-4a5b33c2f826-credential-socket\") on node \"ip-10-0-139-174.ec2.internal\" DevicePath \"\"" Sep 10 13:22:46.760705 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:22:46.760670 2557 generic.go:356] "Generic (PLEG): container finished" podID="30d85e7c-08a7-41a2-beb0-4a5b33c2f826" containerID="08d35bd338141e95053ece69a357e2178eb5f14f2de92dce47691db334eac786" exitCode=0 Sep 10 13:22:46.760869 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:22:46.760724 2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-fddd6c88c-gf8ms" Sep 10 13:22:46.760869 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:22:46.760757 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-fddd6c88c-gf8ms" event={"ID":"30d85e7c-08a7-41a2-beb0-4a5b33c2f826","Type":"ContainerDied","Data":"08d35bd338141e95053ece69a357e2178eb5f14f2de92dce47691db334eac786"} Sep 10 13:22:46.760869 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:22:46.760818 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-fddd6c88c-gf8ms" event={"ID":"30d85e7c-08a7-41a2-beb0-4a5b33c2f826","Type":"ContainerDied","Data":"9423920e60fd48baaed7d5d38a8dfa88c3307886df989701e2c43aab32cb2e9f"} Sep 10 13:22:46.760869 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:22:46.760846 2557 scope.go:117] "RemoveContainer" containerID="08d35bd338141e95053ece69a357e2178eb5f14f2de92dce47691db334eac786" Sep 10 13:22:46.762958 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:22:46.762928 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-tkt56" event={"ID":"d978ed31-3a9b-4efb-b5c9-da53484bebc9","Type":"ContainerStarted","Data":"b7007ac35f0c3532dcafd81b901e0a7f5ee190dead0929955a1480cef932f490"} Sep 10 13:22:46.763082 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:22:46.763039 2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-tkt56" Sep 10 13:22:46.770385 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:22:46.770366 2557 scope.go:117] "RemoveContainer" containerID="08d35bd338141e95053ece69a357e2178eb5f14f2de92dce47691db334eac786" Sep 10 13:22:46.770709 ip-10-0-139-174 kubenswrapper[2557]: E0910 13:22:46.770691 2557 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"08d35bd338141e95053ece69a357e2178eb5f14f2de92dce47691db334eac786\": container with ID starting with 08d35bd338141e95053ece69a357e2178eb5f14f2de92dce47691db334eac786 not found: ID does not exist" containerID="08d35bd338141e95053ece69a357e2178eb5f14f2de92dce47691db334eac786" Sep 10 13:22:46.770757 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:22:46.770717 2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"08d35bd338141e95053ece69a357e2178eb5f14f2de92dce47691db334eac786"} err="failed to get container status \"08d35bd338141e95053ece69a357e2178eb5f14f2de92dce47691db334eac786\": rpc error: code = NotFound desc = could not find container \"08d35bd338141e95053ece69a357e2178eb5f14f2de92dce47691db334eac786\": container with ID starting with 08d35bd338141e95053ece69a357e2178eb5f14f2de92dce47691db334eac786 not found: ID does not exist" Sep 10 13:22:46.807297 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:22:46.807274 2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-fddd6c88c-gf8ms"] Sep 10 13:22:46.818413 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:22:46.818386 2557 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-fddd6c88c-gf8ms"] Sep 10 13:22:46.853578 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:22:46.853259 2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-tkt56" podStartSLOduration=1.665122007 podStartE2EDuration="3.853238019s" totalImagesPullingTime="2.188116012s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 13:22:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 13:22:46.851412151 +0000 UTC m=+327.158406564" watchObservedRunningTime="2026-09-10 13:22:46.853238019 +0000 UTC m=+327.160232436" Sep 10 13:22:47.771093 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:22:47.771056 2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-tkt56" Sep 10 13:22:48.220741 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:22:48.220708 2557 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="30d85e7c-08a7-41a2-beb0-4a5b33c2f826" path="/var/lib/kubelet/pods/30d85e7c-08a7-41a2-beb0-4a5b33c2f826/volumes" Sep 10 13:22:52.737258 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:22:52.737230 2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/odh-model-controller-7998c67dc4-9rt5c" Sep 10 13:22:52.739025 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:22:52.739006 2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/model-serving-api-54d89bd5fc-fc5qw" Sep 10 13:23:42.996455 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:23:42.996404 2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847afpr5j"] Sep 10 13:23:42.997027 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:23:42.996903 2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="30d85e7c-08a7-41a2-beb0-4a5b33c2f826" containerName="istio-proxy" Sep 10 13:23:42.997027 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:23:42.996923 2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="30d85e7c-08a7-41a2-beb0-4a5b33c2f826" containerName="istio-proxy" Sep 10 13:23:42.997027 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:23:42.997025 2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="30d85e7c-08a7-41a2-beb0-4a5b33c2f826" containerName="istio-proxy" Sep 10 13:23:42.999238 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:23:42.999214 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847afpr5j" Sep 10 13:23:43.008494 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:23:43.008473 2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\"" Sep 10 13:23:43.008608 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:23:43.008476 2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\"" Sep 10 13:23:43.008608 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:23:43.008551 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-9r2bh\"" Sep 10 13:23:43.016332 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:23:43.016311 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847afpr5j"] Sep 10 13:23:43.056933 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:23:43.056911 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/37adea46-304c-4af5-8ecc-4c3030f81909-bundle\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847afpr5j\" (UID: \"37adea46-304c-4af5-8ecc-4c3030f81909\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847afpr5j" Sep 10 13:23:43.057071 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:23:43.056946 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m8knq\" (UniqueName: \"kubernetes.io/projected/37adea46-304c-4af5-8ecc-4c3030f81909-kube-api-access-m8knq\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847afpr5j\" (UID: \"37adea46-304c-4af5-8ecc-4c3030f81909\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847afpr5j" Sep 10 13:23:43.057071 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:23:43.056981 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/37adea46-304c-4af5-8ecc-4c3030f81909-util\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847afpr5j\" (UID: \"37adea46-304c-4af5-8ecc-4c3030f81909\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847afpr5j" Sep 10 13:23:43.131216 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:23:43.131181 2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6rb6rm"] Sep 10 13:23:43.134666 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:23:43.134647 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6rb6rm" Sep 10 13:23:43.157836 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:23:43.157814 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/37adea46-304c-4af5-8ecc-4c3030f81909-util\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847afpr5j\" (UID: \"37adea46-304c-4af5-8ecc-4c3030f81909\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847afpr5j" Sep 10 13:23:43.157916 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:23:43.157858 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/86f226bb-e9c5-4073-9652-6b5031f4dd8f-bundle\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6rb6rm\" (UID: \"86f226bb-e9c5-4073-9652-6b5031f4dd8f\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6rb6rm" Sep 10 13:23:43.157916 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:23:43.157888 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-glrzb\" (UniqueName: \"kubernetes.io/projected/86f226bb-e9c5-4073-9652-6b5031f4dd8f-kube-api-access-glrzb\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6rb6rm\" (UID: \"86f226bb-e9c5-4073-9652-6b5031f4dd8f\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6rb6rm" Sep 10 13:23:43.157916 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:23:43.157911 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/37adea46-304c-4af5-8ecc-4c3030f81909-bundle\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847afpr5j\" (UID: \"37adea46-304c-4af5-8ecc-4c3030f81909\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847afpr5j" Sep 10 13:23:43.158051 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:23:43.157933 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m8knq\" (UniqueName: \"kubernetes.io/projected/37adea46-304c-4af5-8ecc-4c3030f81909-kube-api-access-m8knq\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847afpr5j\" (UID: \"37adea46-304c-4af5-8ecc-4c3030f81909\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847afpr5j" Sep 10 13:23:43.158051 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:23:43.157951 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/86f226bb-e9c5-4073-9652-6b5031f4dd8f-util\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6rb6rm\" (UID: \"86f226bb-e9c5-4073-9652-6b5031f4dd8f\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6rb6rm" Sep 10 13:23:43.158194 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:23:43.158174 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/37adea46-304c-4af5-8ecc-4c3030f81909-util\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847afpr5j\" (UID: \"37adea46-304c-4af5-8ecc-4c3030f81909\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847afpr5j" Sep 10 13:23:43.158327 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:23:43.158311 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/37adea46-304c-4af5-8ecc-4c3030f81909-bundle\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847afpr5j\" (UID: \"37adea46-304c-4af5-8ecc-4c3030f81909\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847afpr5j" Sep 10 13:23:43.167853 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:23:43.167815 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6rb6rm"] Sep 10 13:23:43.177544 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:23:43.177522 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-m8knq\" (UniqueName: \"kubernetes.io/projected/37adea46-304c-4af5-8ecc-4c3030f81909-kube-api-access-m8knq\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847afpr5j\" (UID: \"37adea46-304c-4af5-8ecc-4c3030f81909\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847afpr5j" Sep 10 13:23:43.258997 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:23:43.258934 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/86f226bb-e9c5-4073-9652-6b5031f4dd8f-util\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6rb6rm\" (UID: \"86f226bb-e9c5-4073-9652-6b5031f4dd8f\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6rb6rm" Sep 10 13:23:43.259100 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:23:43.259003 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/86f226bb-e9c5-4073-9652-6b5031f4dd8f-bundle\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6rb6rm\" (UID: \"86f226bb-e9c5-4073-9652-6b5031f4dd8f\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6rb6rm" Sep 10 13:23:43.259100 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:23:43.259037 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-glrzb\" (UniqueName: \"kubernetes.io/projected/86f226bb-e9c5-4073-9652-6b5031f4dd8f-kube-api-access-glrzb\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6rb6rm\" (UID: \"86f226bb-e9c5-4073-9652-6b5031f4dd8f\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6rb6rm" Sep 10 13:23:43.259369 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:23:43.259346 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/86f226bb-e9c5-4073-9652-6b5031f4dd8f-util\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6rb6rm\" (UID: \"86f226bb-e9c5-4073-9652-6b5031f4dd8f\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6rb6rm" Sep 10 13:23:43.259414 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:23:43.259357 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/86f226bb-e9c5-4073-9652-6b5031f4dd8f-bundle\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6rb6rm\" (UID: \"86f226bb-e9c5-4073-9652-6b5031f4dd8f\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6rb6rm" Sep 10 13:23:43.260661 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:23:43.260641 2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7blmgj"] Sep 10 13:23:43.263338 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:23:43.263323 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7blmgj" Sep 10 13:23:43.273935 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:23:43.273914 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-glrzb\" (UniqueName: \"kubernetes.io/projected/86f226bb-e9c5-4073-9652-6b5031f4dd8f-kube-api-access-glrzb\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6rb6rm\" (UID: \"86f226bb-e9c5-4073-9652-6b5031f4dd8f\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6rb6rm" Sep 10 13:23:43.281914 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:23:43.281890 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7blmgj"] Sep 10 13:23:43.308343 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:23:43.308318 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847afpr5j" Sep 10 13:23:43.360115 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:23:43.360048 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w8rgb\" (UniqueName: \"kubernetes.io/projected/0f24bc8c-1f1f-48c3-aba5-3afe433db054-kube-api-access-w8rgb\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7blmgj\" (UID: \"0f24bc8c-1f1f-48c3-aba5-3afe433db054\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7blmgj" Sep 10 13:23:43.360249 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:23:43.360167 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/0f24bc8c-1f1f-48c3-aba5-3afe433db054-bundle\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7blmgj\" (UID: \"0f24bc8c-1f1f-48c3-aba5-3afe433db054\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7blmgj" Sep 10 13:23:43.360249 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:23:43.360201 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/0f24bc8c-1f1f-48c3-aba5-3afe433db054-util\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7blmgj\" (UID: \"0f24bc8c-1f1f-48c3-aba5-3afe433db054\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7blmgj" Sep 10 13:23:43.395874 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:23:43.395843 2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cz5gpc"] Sep 10 13:23:43.399278 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:23:43.399261 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cz5gpc" Sep 10 13:23:43.418388 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:23:43.418359 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cz5gpc"] Sep 10 13:23:43.442881 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:23:43.442852 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6rb6rm" Sep 10 13:23:43.460784 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:23:43.460745 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w8rgb\" (UniqueName: \"kubernetes.io/projected/0f24bc8c-1f1f-48c3-aba5-3afe433db054-kube-api-access-w8rgb\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7blmgj\" (UID: \"0f24bc8c-1f1f-48c3-aba5-3afe433db054\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7blmgj" Sep 10 13:23:43.460905 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:23:43.460814 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9trfk\" (UniqueName: \"kubernetes.io/projected/b9879932-0222-4f35-b29c-e75b5b8619d4-kube-api-access-9trfk\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cz5gpc\" (UID: \"b9879932-0222-4f35-b29c-e75b5b8619d4\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cz5gpc" Sep 10 13:23:43.460905 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:23:43.460848 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/b9879932-0222-4f35-b29c-e75b5b8619d4-util\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cz5gpc\" (UID: \"b9879932-0222-4f35-b29c-e75b5b8619d4\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cz5gpc" Sep 10 13:23:43.461015 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:23:43.460910 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/0f24bc8c-1f1f-48c3-aba5-3afe433db054-bundle\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7blmgj\" (UID: \"0f24bc8c-1f1f-48c3-aba5-3afe433db054\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7blmgj" Sep 10 13:23:43.461015 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:23:43.460949 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/b9879932-0222-4f35-b29c-e75b5b8619d4-bundle\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cz5gpc\" (UID: \"b9879932-0222-4f35-b29c-e75b5b8619d4\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cz5gpc" Sep 10 13:23:43.461015 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:23:43.460973 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/0f24bc8c-1f1f-48c3-aba5-3afe433db054-util\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7blmgj\" (UID: \"0f24bc8c-1f1f-48c3-aba5-3afe433db054\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7blmgj" Sep 10 13:23:43.461364 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:23:43.461343 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/0f24bc8c-1f1f-48c3-aba5-3afe433db054-bundle\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7blmgj\" (UID: \"0f24bc8c-1f1f-48c3-aba5-3afe433db054\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7blmgj" Sep 10 13:23:43.461364 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:23:43.461354 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/0f24bc8c-1f1f-48c3-aba5-3afe433db054-util\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7blmgj\" (UID: \"0f24bc8c-1f1f-48c3-aba5-3afe433db054\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7blmgj" Sep 10 13:23:43.473584 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:23:43.473559 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-w8rgb\" (UniqueName: \"kubernetes.io/projected/0f24bc8c-1f1f-48c3-aba5-3afe433db054-kube-api-access-w8rgb\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7blmgj\" (UID: \"0f24bc8c-1f1f-48c3-aba5-3afe433db054\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7blmgj" Sep 10 13:23:43.561647 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:23:43.561578 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9trfk\" (UniqueName: \"kubernetes.io/projected/b9879932-0222-4f35-b29c-e75b5b8619d4-kube-api-access-9trfk\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cz5gpc\" (UID: \"b9879932-0222-4f35-b29c-e75b5b8619d4\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cz5gpc" Sep 10 13:23:43.561647 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:23:43.561616 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/b9879932-0222-4f35-b29c-e75b5b8619d4-util\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cz5gpc\" (UID: \"b9879932-0222-4f35-b29c-e75b5b8619d4\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cz5gpc" Sep 10 13:23:43.561805 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:23:43.561649 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/b9879932-0222-4f35-b29c-e75b5b8619d4-bundle\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cz5gpc\" (UID: \"b9879932-0222-4f35-b29c-e75b5b8619d4\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cz5gpc" Sep 10 13:23:43.562037 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:23:43.562015 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/b9879932-0222-4f35-b29c-e75b5b8619d4-util\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cz5gpc\" (UID: \"b9879932-0222-4f35-b29c-e75b5b8619d4\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cz5gpc" Sep 10 13:23:43.562078 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:23:43.562041 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/b9879932-0222-4f35-b29c-e75b5b8619d4-bundle\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cz5gpc\" (UID: \"b9879932-0222-4f35-b29c-e75b5b8619d4\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cz5gpc" Sep 10 13:23:43.571831 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:23:43.571797 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7blmgj" Sep 10 13:23:43.575805 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:23:43.575783 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9trfk\" (UniqueName: \"kubernetes.io/projected/b9879932-0222-4f35-b29c-e75b5b8619d4-kube-api-access-9trfk\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cz5gpc\" (UID: \"b9879932-0222-4f35-b29c-e75b5b8619d4\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cz5gpc" Sep 10 13:23:43.606171 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:23:43.606143 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847afpr5j"] Sep 10 13:23:43.709296 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:23:43.709266 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cz5gpc" Sep 10 13:23:43.733579 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:23:43.733557 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6rb6rm"] Sep 10 13:23:43.770364 ip-10-0-139-174 kubenswrapper[2557]: W0910 13:23:43.770333 2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod86f226bb_e9c5_4073_9652_6b5031f4dd8f.slice/crio-11ab1a6fd523ed311b37aae0fc0d9b828018bf73f4b7b6972a571e17ab5aebd6 WatchSource:0}: Error finding container 11ab1a6fd523ed311b37aae0fc0d9b828018bf73f4b7b6972a571e17ab5aebd6: Status 404 returned error can't find the container with id 11ab1a6fd523ed311b37aae0fc0d9b828018bf73f4b7b6972a571e17ab5aebd6 Sep 10 13:23:43.861681 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:23:43.861648 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7blmgj"] Sep 10 13:23:43.899165 ip-10-0-139-174 kubenswrapper[2557]: W0910 13:23:43.899137 2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0f24bc8c_1f1f_48c3_aba5_3afe433db054.slice/crio-f89e183ecc1d3d8349879d51b7fad25fe9b6b1d8fafd8c0d95336a044b6a9d9f WatchSource:0}: Error finding container f89e183ecc1d3d8349879d51b7fad25fe9b6b1d8fafd8c0d95336a044b6a9d9f: Status 404 returned error can't find the container with id f89e183ecc1d3d8349879d51b7fad25fe9b6b1d8fafd8c0d95336a044b6a9d9f Sep 10 13:23:43.995009 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:23:43.994984 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cz5gpc"] Sep 10 13:23:44.042864 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:23:44.042841 2557 generic.go:356] "Generic (PLEG): container finished" podID="86f226bb-e9c5-4073-9652-6b5031f4dd8f" containerID="9d2621340fb3afaa8b5a26feae3a52158dee5eedf1e75cf9c3158883c00c3b61" exitCode=0 Sep 10 13:23:44.043171 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:23:44.042926 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6rb6rm" event={"ID":"86f226bb-e9c5-4073-9652-6b5031f4dd8f","Type":"ContainerDied","Data":"9d2621340fb3afaa8b5a26feae3a52158dee5eedf1e75cf9c3158883c00c3b61"} Sep 10 13:23:44.043171 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:23:44.042977 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6rb6rm" event={"ID":"86f226bb-e9c5-4073-9652-6b5031f4dd8f","Type":"ContainerStarted","Data":"11ab1a6fd523ed311b37aae0fc0d9b828018bf73f4b7b6972a571e17ab5aebd6"} Sep 10 13:23:44.044709 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:23:44.044674 2557 generic.go:356] "Generic (PLEG): container finished" podID="0f24bc8c-1f1f-48c3-aba5-3afe433db054" containerID="5201e4a52c96c1e1f164798db956a5d31d8191ffad272953406389c6165921e7" exitCode=0 Sep 10 13:23:44.044789 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:23:44.044762 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7blmgj" event={"ID":"0f24bc8c-1f1f-48c3-aba5-3afe433db054","Type":"ContainerDied","Data":"5201e4a52c96c1e1f164798db956a5d31d8191ffad272953406389c6165921e7"} Sep 10 13:23:44.044852 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:23:44.044807 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7blmgj" event={"ID":"0f24bc8c-1f1f-48c3-aba5-3afe433db054","Type":"ContainerStarted","Data":"f89e183ecc1d3d8349879d51b7fad25fe9b6b1d8fafd8c0d95336a044b6a9d9f"} Sep 10 13:23:44.046356 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:23:44.046339 2557 generic.go:356] "Generic (PLEG): container finished" podID="37adea46-304c-4af5-8ecc-4c3030f81909" containerID="79e249af46031e886ef104bfdec0963411bda071e49c2dad2f1755b2e3d23e3e" exitCode=0 Sep 10 13:23:44.046457 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:23:44.046371 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847afpr5j" event={"ID":"37adea46-304c-4af5-8ecc-4c3030f81909","Type":"ContainerDied","Data":"79e249af46031e886ef104bfdec0963411bda071e49c2dad2f1755b2e3d23e3e"} Sep 10 13:23:44.046457 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:23:44.046405 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847afpr5j" event={"ID":"37adea46-304c-4af5-8ecc-4c3030f81909","Type":"ContainerStarted","Data":"abfa7e251fea85498fd0c4f90ff364e38fc315b096b1b68903cc454374ba0659"} Sep 10 13:23:44.059141 ip-10-0-139-174 kubenswrapper[2557]: W0910 13:23:44.059122 2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb9879932_0222_4f35_b29c_e75b5b8619d4.slice/crio-f9b0fbe5d8407e413200b17abaf6889ecca15199ec64d90a3d3bd772d2028534 WatchSource:0}: Error finding container f9b0fbe5d8407e413200b17abaf6889ecca15199ec64d90a3d3bd772d2028534: Status 404 returned error can't find the container with id f9b0fbe5d8407e413200b17abaf6889ecca15199ec64d90a3d3bd772d2028534 Sep 10 13:23:45.052252 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:23:45.052222 2557 generic.go:356] "Generic (PLEG): container finished" podID="b9879932-0222-4f35-b29c-e75b5b8619d4" containerID="3d1bdfafb73b9d2d4715c78e4d51ed48a851cada20640716f7c03bf18e4b89b9" exitCode=0 Sep 10 13:23:45.052585 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:23:45.052295 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cz5gpc" event={"ID":"b9879932-0222-4f35-b29c-e75b5b8619d4","Type":"ContainerDied","Data":"3d1bdfafb73b9d2d4715c78e4d51ed48a851cada20640716f7c03bf18e4b89b9"} Sep 10 13:23:45.052585 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:23:45.052379 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cz5gpc" event={"ID":"b9879932-0222-4f35-b29c-e75b5b8619d4","Type":"ContainerStarted","Data":"f9b0fbe5d8407e413200b17abaf6889ecca15199ec64d90a3d3bd772d2028534"} Sep 10 13:23:45.054224 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:23:45.054207 2557 generic.go:356] "Generic (PLEG): container finished" podID="0f24bc8c-1f1f-48c3-aba5-3afe433db054" containerID="0956b3f80f6cf38880586b6454389b7feda46ca59394a761efaea96d03877e64" exitCode=0 Sep 10 13:23:45.054323 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:23:45.054250 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7blmgj" event={"ID":"0f24bc8c-1f1f-48c3-aba5-3afe433db054","Type":"ContainerDied","Data":"0956b3f80f6cf38880586b6454389b7feda46ca59394a761efaea96d03877e64"} Sep 10 13:23:46.062071 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:23:46.062036 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cz5gpc" event={"ID":"b9879932-0222-4f35-b29c-e75b5b8619d4","Type":"ContainerStarted","Data":"2614b448213211173cffe54f366d8bf131fd58c8e0d2179871ae7d4c169b646e"} Sep 10 13:23:46.064390 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:23:46.064367 2557 generic.go:356] "Generic (PLEG): container finished" podID="0f24bc8c-1f1f-48c3-aba5-3afe433db054" containerID="395e4dd79b63a9cabab8d274c9b9b7fc349885045315daddc88c5297ab6e1b24" exitCode=0 Sep 10 13:23:46.064561 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:23:46.064446 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7blmgj" event={"ID":"0f24bc8c-1f1f-48c3-aba5-3afe433db054","Type":"ContainerDied","Data":"395e4dd79b63a9cabab8d274c9b9b7fc349885045315daddc88c5297ab6e1b24"} Sep 10 13:23:47.071606 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:23:47.071572 2557 generic.go:356] "Generic (PLEG): container finished" podID="37adea46-304c-4af5-8ecc-4c3030f81909" containerID="de0a07b6c32b30298e61b1a8c91caccf84d99e5b489dde66d4485b236a85fd41" exitCode=0 Sep 10 13:23:47.072018 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:23:47.071671 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847afpr5j" event={"ID":"37adea46-304c-4af5-8ecc-4c3030f81909","Type":"ContainerDied","Data":"de0a07b6c32b30298e61b1a8c91caccf84d99e5b489dde66d4485b236a85fd41"} Sep 10 13:23:47.073722 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:23:47.073704 2557 generic.go:356] "Generic (PLEG): container finished" podID="b9879932-0222-4f35-b29c-e75b5b8619d4" containerID="2614b448213211173cffe54f366d8bf131fd58c8e0d2179871ae7d4c169b646e" exitCode=0 Sep 10 13:23:47.073806 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:23:47.073785 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cz5gpc" event={"ID":"b9879932-0222-4f35-b29c-e75b5b8619d4","Type":"ContainerDied","Data":"2614b448213211173cffe54f366d8bf131fd58c8e0d2179871ae7d4c169b646e"} Sep 10 13:23:47.208875 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:23:47.208847 2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7blmgj" Sep 10 13:23:47.291178 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:23:47.291148 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-w8rgb\" (UniqueName: \"kubernetes.io/projected/0f24bc8c-1f1f-48c3-aba5-3afe433db054-kube-api-access-w8rgb\") pod \"0f24bc8c-1f1f-48c3-aba5-3afe433db054\" (UID: \"0f24bc8c-1f1f-48c3-aba5-3afe433db054\") " Sep 10 13:23:47.291313 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:23:47.291208 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/0f24bc8c-1f1f-48c3-aba5-3afe433db054-util\") pod \"0f24bc8c-1f1f-48c3-aba5-3afe433db054\" (UID: \"0f24bc8c-1f1f-48c3-aba5-3afe433db054\") " Sep 10 13:23:47.291313 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:23:47.291227 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/0f24bc8c-1f1f-48c3-aba5-3afe433db054-bundle\") pod \"0f24bc8c-1f1f-48c3-aba5-3afe433db054\" (UID: \"0f24bc8c-1f1f-48c3-aba5-3afe433db054\") " Sep 10 13:23:47.291816 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:23:47.291786 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0f24bc8c-1f1f-48c3-aba5-3afe433db054-bundle" (OuterVolumeSpecName: "bundle") pod "0f24bc8c-1f1f-48c3-aba5-3afe433db054" (UID: "0f24bc8c-1f1f-48c3-aba5-3afe433db054"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 13:23:47.293386 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:23:47.293367 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0f24bc8c-1f1f-48c3-aba5-3afe433db054-kube-api-access-w8rgb" (OuterVolumeSpecName: "kube-api-access-w8rgb") pod "0f24bc8c-1f1f-48c3-aba5-3afe433db054" (UID: "0f24bc8c-1f1f-48c3-aba5-3afe433db054"). InnerVolumeSpecName "kube-api-access-w8rgb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 10 13:23:47.296779 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:23:47.296759 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0f24bc8c-1f1f-48c3-aba5-3afe433db054-util" (OuterVolumeSpecName: "util") pod "0f24bc8c-1f1f-48c3-aba5-3afe433db054" (UID: "0f24bc8c-1f1f-48c3-aba5-3afe433db054"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 13:23:47.392467 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:23:47.392445 2557 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w8rgb\" (UniqueName: \"kubernetes.io/projected/0f24bc8c-1f1f-48c3-aba5-3afe433db054-kube-api-access-w8rgb\") on node \"ip-10-0-139-174.ec2.internal\" DevicePath \"\"" Sep 10 13:23:47.392557 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:23:47.392472 2557 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/0f24bc8c-1f1f-48c3-aba5-3afe433db054-util\") on node \"ip-10-0-139-174.ec2.internal\" DevicePath \"\"" Sep 10 13:23:47.392557 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:23:47.392487 2557 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/0f24bc8c-1f1f-48c3-aba5-3afe433db054-bundle\") on node \"ip-10-0-139-174.ec2.internal\" DevicePath \"\"" Sep 10 13:23:48.087194 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:23:48.087162 2557 generic.go:356] "Generic (PLEG): container finished" podID="86f226bb-e9c5-4073-9652-6b5031f4dd8f" containerID="e41d18b49a903ef5527193e244ac25188fb07754d89bb33b2b3d289fb3cc490d" exitCode=0 Sep 10 13:23:48.087631 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:23:48.087248 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6rb6rm" event={"ID":"86f226bb-e9c5-4073-9652-6b5031f4dd8f","Type":"ContainerDied","Data":"e41d18b49a903ef5527193e244ac25188fb07754d89bb33b2b3d289fb3cc490d"} Sep 10 13:23:48.089462 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:23:48.089442 2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7blmgj" Sep 10 13:23:48.089462 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:23:48.089451 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7blmgj" event={"ID":"0f24bc8c-1f1f-48c3-aba5-3afe433db054","Type":"ContainerDied","Data":"f89e183ecc1d3d8349879d51b7fad25fe9b6b1d8fafd8c0d95336a044b6a9d9f"} Sep 10 13:23:48.089616 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:23:48.089491 2557 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f89e183ecc1d3d8349879d51b7fad25fe9b6b1d8fafd8c0d95336a044b6a9d9f" Sep 10 13:23:48.092151 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:23:48.092126 2557 generic.go:356] "Generic (PLEG): container finished" podID="37adea46-304c-4af5-8ecc-4c3030f81909" containerID="c3e9766a61fc339ae7c4ea7e18694c302e85b94b819b82580aea7726afe85e44" exitCode=0 Sep 10 13:23:48.092280 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:23:48.092205 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847afpr5j" event={"ID":"37adea46-304c-4af5-8ecc-4c3030f81909","Type":"ContainerDied","Data":"c3e9766a61fc339ae7c4ea7e18694c302e85b94b819b82580aea7726afe85e44"} Sep 10 13:23:48.094806 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:23:48.094785 2557 generic.go:356] "Generic (PLEG): container finished" podID="b9879932-0222-4f35-b29c-e75b5b8619d4" containerID="9d6c443dc2a23f18c0a4848d4f90a5aa4a7844329ad954e4c145c80a566bbf5c" exitCode=0 Sep 10 13:23:48.094883 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:23:48.094840 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cz5gpc" event={"ID":"b9879932-0222-4f35-b29c-e75b5b8619d4","Type":"ContainerDied","Data":"9d6c443dc2a23f18c0a4848d4f90a5aa4a7844329ad954e4c145c80a566bbf5c"} Sep 10 13:23:49.102295 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:23:49.102240 2557 generic.go:356] "Generic (PLEG): container finished" podID="86f226bb-e9c5-4073-9652-6b5031f4dd8f" containerID="d1a1709eba6f8700b25435d44541fad275a3e98f5517e85cab035a3ba87709ac" exitCode=0 Sep 10 13:23:49.102677 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:23:49.102327 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6rb6rm" event={"ID":"86f226bb-e9c5-4073-9652-6b5031f4dd8f","Type":"ContainerDied","Data":"d1a1709eba6f8700b25435d44541fad275a3e98f5517e85cab035a3ba87709ac"} Sep 10 13:23:49.255917 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:23:49.255896 2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cz5gpc" Sep 10 13:23:49.258938 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:23:49.258921 2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847afpr5j" Sep 10 13:23:49.307659 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:23:49.307636 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-m8knq\" (UniqueName: \"kubernetes.io/projected/37adea46-304c-4af5-8ecc-4c3030f81909-kube-api-access-m8knq\") pod \"37adea46-304c-4af5-8ecc-4c3030f81909\" (UID: \"37adea46-304c-4af5-8ecc-4c3030f81909\") " Sep 10 13:23:49.307784 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:23:49.307691 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/b9879932-0222-4f35-b29c-e75b5b8619d4-bundle\") pod \"b9879932-0222-4f35-b29c-e75b5b8619d4\" (UID: \"b9879932-0222-4f35-b29c-e75b5b8619d4\") " Sep 10 13:23:49.307784 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:23:49.307731 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9trfk\" (UniqueName: \"kubernetes.io/projected/b9879932-0222-4f35-b29c-e75b5b8619d4-kube-api-access-9trfk\") pod \"b9879932-0222-4f35-b29c-e75b5b8619d4\" (UID: \"b9879932-0222-4f35-b29c-e75b5b8619d4\") " Sep 10 13:23:49.307784 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:23:49.307773 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/b9879932-0222-4f35-b29c-e75b5b8619d4-util\") pod \"b9879932-0222-4f35-b29c-e75b5b8619d4\" (UID: \"b9879932-0222-4f35-b29c-e75b5b8619d4\") " Sep 10 13:23:49.307916 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:23:49.307804 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/37adea46-304c-4af5-8ecc-4c3030f81909-bundle\") pod \"37adea46-304c-4af5-8ecc-4c3030f81909\" (UID: \"37adea46-304c-4af5-8ecc-4c3030f81909\") " Sep 10 13:23:49.307916 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:23:49.307846 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/37adea46-304c-4af5-8ecc-4c3030f81909-util\") pod \"37adea46-304c-4af5-8ecc-4c3030f81909\" (UID: \"37adea46-304c-4af5-8ecc-4c3030f81909\") " Sep 10 13:23:49.308673 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:23:49.308645 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b9879932-0222-4f35-b29c-e75b5b8619d4-bundle" (OuterVolumeSpecName: "bundle") pod "b9879932-0222-4f35-b29c-e75b5b8619d4" (UID: "b9879932-0222-4f35-b29c-e75b5b8619d4"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 13:23:49.308761 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:23:49.308664 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/37adea46-304c-4af5-8ecc-4c3030f81909-bundle" (OuterVolumeSpecName: "bundle") pod "37adea46-304c-4af5-8ecc-4c3030f81909" (UID: "37adea46-304c-4af5-8ecc-4c3030f81909"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 13:23:49.309979 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:23:49.309957 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b9879932-0222-4f35-b29c-e75b5b8619d4-kube-api-access-9trfk" (OuterVolumeSpecName: "kube-api-access-9trfk") pod "b9879932-0222-4f35-b29c-e75b5b8619d4" (UID: "b9879932-0222-4f35-b29c-e75b5b8619d4"). InnerVolumeSpecName "kube-api-access-9trfk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 10 13:23:49.310156 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:23:49.310138 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/37adea46-304c-4af5-8ecc-4c3030f81909-kube-api-access-m8knq" (OuterVolumeSpecName: "kube-api-access-m8knq") pod "37adea46-304c-4af5-8ecc-4c3030f81909" (UID: "37adea46-304c-4af5-8ecc-4c3030f81909"). InnerVolumeSpecName "kube-api-access-m8knq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 10 13:23:49.314275 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:23:49.314251 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/37adea46-304c-4af5-8ecc-4c3030f81909-util" (OuterVolumeSpecName: "util") pod "37adea46-304c-4af5-8ecc-4c3030f81909" (UID: "37adea46-304c-4af5-8ecc-4c3030f81909"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 13:23:49.317151 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:23:49.317127 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b9879932-0222-4f35-b29c-e75b5b8619d4-util" (OuterVolumeSpecName: "util") pod "b9879932-0222-4f35-b29c-e75b5b8619d4" (UID: "b9879932-0222-4f35-b29c-e75b5b8619d4"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 13:23:49.408976 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:23:49.408920 2557 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/b9879932-0222-4f35-b29c-e75b5b8619d4-bundle\") on node \"ip-10-0-139-174.ec2.internal\" DevicePath \"\"" Sep 10 13:23:49.408976 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:23:49.408941 2557 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9trfk\" (UniqueName: \"kubernetes.io/projected/b9879932-0222-4f35-b29c-e75b5b8619d4-kube-api-access-9trfk\") on node \"ip-10-0-139-174.ec2.internal\" DevicePath \"\"" Sep 10 13:23:49.408976 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:23:49.408950 2557 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/b9879932-0222-4f35-b29c-e75b5b8619d4-util\") on node \"ip-10-0-139-174.ec2.internal\" DevicePath \"\"" Sep 10 13:23:49.408976 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:23:49.408959 2557 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/37adea46-304c-4af5-8ecc-4c3030f81909-bundle\") on node \"ip-10-0-139-174.ec2.internal\" DevicePath \"\"" Sep 10 13:23:49.408976 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:23:49.408968 2557 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/37adea46-304c-4af5-8ecc-4c3030f81909-util\") on node \"ip-10-0-139-174.ec2.internal\" DevicePath \"\"" Sep 10 13:23:49.408976 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:23:49.408975 2557 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m8knq\" (UniqueName: \"kubernetes.io/projected/37adea46-304c-4af5-8ecc-4c3030f81909-kube-api-access-m8knq\") on node \"ip-10-0-139-174.ec2.internal\" DevicePath \"\"" Sep 10 13:23:50.111567 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:23:50.111539 2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847afpr5j" Sep 10 13:23:50.112032 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:23:50.111564 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847afpr5j" event={"ID":"37adea46-304c-4af5-8ecc-4c3030f81909","Type":"ContainerDied","Data":"abfa7e251fea85498fd0c4f90ff364e38fc315b096b1b68903cc454374ba0659"} Sep 10 13:23:50.112032 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:23:50.111608 2557 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="abfa7e251fea85498fd0c4f90ff364e38fc315b096b1b68903cc454374ba0659" Sep 10 13:23:50.113539 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:23:50.113507 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cz5gpc" event={"ID":"b9879932-0222-4f35-b29c-e75b5b8619d4","Type":"ContainerDied","Data":"f9b0fbe5d8407e413200b17abaf6889ecca15199ec64d90a3d3bd772d2028534"} Sep 10 13:23:50.113660 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:23:50.113557 2557 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f9b0fbe5d8407e413200b17abaf6889ecca15199ec64d90a3d3bd772d2028534" Sep 10 13:23:50.113660 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:23:50.113524 2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cz5gpc" Sep 10 13:23:50.243927 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:23:50.243905 2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6rb6rm" Sep 10 13:23:50.316283 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:23:50.316253 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/86f226bb-e9c5-4073-9652-6b5031f4dd8f-util\") pod \"86f226bb-e9c5-4073-9652-6b5031f4dd8f\" (UID: \"86f226bb-e9c5-4073-9652-6b5031f4dd8f\") " Sep 10 13:23:50.316445 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:23:50.316308 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/86f226bb-e9c5-4073-9652-6b5031f4dd8f-bundle\") pod \"86f226bb-e9c5-4073-9652-6b5031f4dd8f\" (UID: \"86f226bb-e9c5-4073-9652-6b5031f4dd8f\") " Sep 10 13:23:50.316445 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:23:50.316338 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-glrzb\" (UniqueName: \"kubernetes.io/projected/86f226bb-e9c5-4073-9652-6b5031f4dd8f-kube-api-access-glrzb\") pod \"86f226bb-e9c5-4073-9652-6b5031f4dd8f\" (UID: \"86f226bb-e9c5-4073-9652-6b5031f4dd8f\") " Sep 10 13:23:50.316727 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:23:50.316705 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/86f226bb-e9c5-4073-9652-6b5031f4dd8f-bundle" (OuterVolumeSpecName: "bundle") pod "86f226bb-e9c5-4073-9652-6b5031f4dd8f" (UID: "86f226bb-e9c5-4073-9652-6b5031f4dd8f"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 13:23:50.318710 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:23:50.318686 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/86f226bb-e9c5-4073-9652-6b5031f4dd8f-kube-api-access-glrzb" (OuterVolumeSpecName: "kube-api-access-glrzb") pod "86f226bb-e9c5-4073-9652-6b5031f4dd8f" (UID: "86f226bb-e9c5-4073-9652-6b5031f4dd8f"). InnerVolumeSpecName "kube-api-access-glrzb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 10 13:23:50.320731 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:23:50.320708 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/86f226bb-e9c5-4073-9652-6b5031f4dd8f-util" (OuterVolumeSpecName: "util") pod "86f226bb-e9c5-4073-9652-6b5031f4dd8f" (UID: "86f226bb-e9c5-4073-9652-6b5031f4dd8f"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 13:23:50.417131 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:23:50.417078 2557 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-glrzb\" (UniqueName: \"kubernetes.io/projected/86f226bb-e9c5-4073-9652-6b5031f4dd8f-kube-api-access-glrzb\") on node \"ip-10-0-139-174.ec2.internal\" DevicePath \"\"" Sep 10 13:23:50.417131 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:23:50.417099 2557 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/86f226bb-e9c5-4073-9652-6b5031f4dd8f-util\") on node \"ip-10-0-139-174.ec2.internal\" DevicePath \"\"" Sep 10 13:23:50.417131 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:23:50.417108 2557 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/86f226bb-e9c5-4073-9652-6b5031f4dd8f-bundle\") on node \"ip-10-0-139-174.ec2.internal\" DevicePath \"\"" Sep 10 13:23:51.120346 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:23:51.120316 2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6rb6rm" Sep 10 13:23:51.120791 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:23:51.120347 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6rb6rm" event={"ID":"86f226bb-e9c5-4073-9652-6b5031f4dd8f","Type":"ContainerDied","Data":"11ab1a6fd523ed311b37aae0fc0d9b828018bf73f4b7b6972a571e17ab5aebd6"} Sep 10 13:23:51.120791 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:23:51.120448 2557 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="11ab1a6fd523ed311b37aae0fc0d9b828018bf73f4b7b6972a571e17ab5aebd6" Sep 10 13:24:50.418445 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:24:50.418336 2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-hc5bh"] Sep 10 13:24:50.418947 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:24:50.418926 2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b9879932-0222-4f35-b29c-e75b5b8619d4" containerName="util" Sep 10 13:24:50.419002 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:24:50.418954 2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="b9879932-0222-4f35-b29c-e75b5b8619d4" containerName="util" Sep 10 13:24:50.419002 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:24:50.418968 2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="86f226bb-e9c5-4073-9652-6b5031f4dd8f" containerName="pull" Sep 10 13:24:50.419002 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:24:50.418976 2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="86f226bb-e9c5-4073-9652-6b5031f4dd8f" containerName="pull" Sep 10 13:24:50.419002 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:24:50.418992 2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="37adea46-304c-4af5-8ecc-4c3030f81909" containerName="pull" Sep 10 13:24:50.419002 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:24:50.419001 2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="37adea46-304c-4af5-8ecc-4c3030f81909" containerName="pull" Sep 10 13:24:50.419147 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:24:50.419023 2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b9879932-0222-4f35-b29c-e75b5b8619d4" containerName="pull" Sep 10 13:24:50.419147 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:24:50.419031 2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="b9879932-0222-4f35-b29c-e75b5b8619d4" containerName="pull" Sep 10 13:24:50.419147 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:24:50.419042 2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0f24bc8c-1f1f-48c3-aba5-3afe433db054" containerName="extract" Sep 10 13:24:50.419147 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:24:50.419050 2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="0f24bc8c-1f1f-48c3-aba5-3afe433db054" containerName="extract" Sep 10 13:24:50.419147 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:24:50.419058 2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b9879932-0222-4f35-b29c-e75b5b8619d4" containerName="extract" Sep 10 13:24:50.419147 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:24:50.419066 2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="b9879932-0222-4f35-b29c-e75b5b8619d4" containerName="extract" Sep 10 13:24:50.419147 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:24:50.419078 2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="86f226bb-e9c5-4073-9652-6b5031f4dd8f" containerName="util" Sep 10 13:24:50.419147 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:24:50.419085 2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="86f226bb-e9c5-4073-9652-6b5031f4dd8f" containerName="util" Sep 10 13:24:50.419147 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:24:50.419093 2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="86f226bb-e9c5-4073-9652-6b5031f4dd8f" containerName="extract" Sep 10 13:24:50.419147 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:24:50.419101 2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="86f226bb-e9c5-4073-9652-6b5031f4dd8f" containerName="extract" Sep 10 13:24:50.419147 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:24:50.419117 2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="37adea46-304c-4af5-8ecc-4c3030f81909" containerName="util" Sep 10 13:24:50.419147 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:24:50.419125 2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="37adea46-304c-4af5-8ecc-4c3030f81909" containerName="util" Sep 10 13:24:50.419147 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:24:50.419135 2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0f24bc8c-1f1f-48c3-aba5-3afe433db054" containerName="util" Sep 10 13:24:50.419147 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:24:50.419143 2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="0f24bc8c-1f1f-48c3-aba5-3afe433db054" containerName="util" Sep 10 13:24:50.419576 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:24:50.419152 2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0f24bc8c-1f1f-48c3-aba5-3afe433db054" containerName="pull" Sep 10 13:24:50.419576 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:24:50.419160 2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="0f24bc8c-1f1f-48c3-aba5-3afe433db054" containerName="pull" Sep 10 13:24:50.419576 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:24:50.419169 2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="37adea46-304c-4af5-8ecc-4c3030f81909" containerName="extract" Sep 10 13:24:50.419576 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:24:50.419177 2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="37adea46-304c-4af5-8ecc-4c3030f81909" containerName="extract" Sep 10 13:24:50.419576 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:24:50.419274 2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="37adea46-304c-4af5-8ecc-4c3030f81909" containerName="extract" Sep 10 13:24:50.419576 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:24:50.419288 2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="86f226bb-e9c5-4073-9652-6b5031f4dd8f" containerName="extract" Sep 10 13:24:50.419576 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:24:50.419300 2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="b9879932-0222-4f35-b29c-e75b5b8619d4" containerName="extract" Sep 10 13:24:50.419576 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:24:50.419309 2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="0f24bc8c-1f1f-48c3-aba5-3afe433db054" containerName="extract" Sep 10 13:24:50.422572 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:24:50.422552 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-hc5bh" Sep 10 13:24:50.424743 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:24:50.424723 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"maas-default-gateway-openshift-default-dockercfg-8zz22\"" Sep 10 13:24:50.434800 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:24:50.434777 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-hc5bh"] Sep 10 13:24:50.493318 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:24:50.493287 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c933115e-f7d6-4878-8d5f-a4c81447652d-istio-podinfo\") pod \"maas-default-gateway-openshift-default-5ccb968c66-hc5bh\" (UID: \"c933115e-f7d6-4878-8d5f-a4c81447652d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-hc5bh" Sep 10 13:24:50.493485 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:24:50.493326 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c933115e-f7d6-4878-8d5f-a4c81447652d-credential-socket\") pod \"maas-default-gateway-openshift-default-5ccb968c66-hc5bh\" (UID: \"c933115e-f7d6-4878-8d5f-a4c81447652d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-hc5bh" Sep 10 13:24:50.493485 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:24:50.493350 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c933115e-f7d6-4878-8d5f-a4c81447652d-istio-token\") pod \"maas-default-gateway-openshift-default-5ccb968c66-hc5bh\" (UID: \"c933115e-f7d6-4878-8d5f-a4c81447652d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-hc5bh" Sep 10 13:24:50.493485 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:24:50.493368 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c933115e-f7d6-4878-8d5f-a4c81447652d-istio-data\") pod \"maas-default-gateway-openshift-default-5ccb968c66-hc5bh\" (UID: \"c933115e-f7d6-4878-8d5f-a4c81447652d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-hc5bh" Sep 10 13:24:50.493485 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:24:50.493446 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c933115e-f7d6-4878-8d5f-a4c81447652d-istio-envoy\") pod \"maas-default-gateway-openshift-default-5ccb968c66-hc5bh\" (UID: \"c933115e-f7d6-4878-8d5f-a4c81447652d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-hc5bh" Sep 10 13:24:50.493485 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:24:50.493461 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c933115e-f7d6-4878-8d5f-a4c81447652d-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-5ccb968c66-hc5bh\" (UID: \"c933115e-f7d6-4878-8d5f-a4c81447652d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-hc5bh" Sep 10 13:24:50.493485 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:24:50.493479 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pm5mj\" (UniqueName: \"kubernetes.io/projected/c933115e-f7d6-4878-8d5f-a4c81447652d-kube-api-access-pm5mj\") pod \"maas-default-gateway-openshift-default-5ccb968c66-hc5bh\" (UID: \"c933115e-f7d6-4878-8d5f-a4c81447652d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-hc5bh" Sep 10 13:24:50.493690 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:24:50.493503 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c933115e-f7d6-4878-8d5f-a4c81447652d-workload-certs\") pod \"maas-default-gateway-openshift-default-5ccb968c66-hc5bh\" (UID: \"c933115e-f7d6-4878-8d5f-a4c81447652d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-hc5bh" Sep 10 13:24:50.493690 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:24:50.493534 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c933115e-f7d6-4878-8d5f-a4c81447652d-workload-socket\") pod \"maas-default-gateway-openshift-default-5ccb968c66-hc5bh\" (UID: \"c933115e-f7d6-4878-8d5f-a4c81447652d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-hc5bh" Sep 10 13:24:50.594331 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:24:50.594297 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c933115e-f7d6-4878-8d5f-a4c81447652d-workload-certs\") pod \"maas-default-gateway-openshift-default-5ccb968c66-hc5bh\" (UID: \"c933115e-f7d6-4878-8d5f-a4c81447652d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-hc5bh" Sep 10 13:24:50.594514 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:24:50.594354 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c933115e-f7d6-4878-8d5f-a4c81447652d-workload-socket\") pod \"maas-default-gateway-openshift-default-5ccb968c66-hc5bh\" (UID: \"c933115e-f7d6-4878-8d5f-a4c81447652d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-hc5bh" Sep 10 13:24:50.594514 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:24:50.594386 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c933115e-f7d6-4878-8d5f-a4c81447652d-istio-podinfo\") pod \"maas-default-gateway-openshift-default-5ccb968c66-hc5bh\" (UID: \"c933115e-f7d6-4878-8d5f-a4c81447652d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-hc5bh" Sep 10 13:24:50.594514 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:24:50.594438 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c933115e-f7d6-4878-8d5f-a4c81447652d-credential-socket\") pod \"maas-default-gateway-openshift-default-5ccb968c66-hc5bh\" (UID: \"c933115e-f7d6-4878-8d5f-a4c81447652d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-hc5bh" Sep 10 13:24:50.594514 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:24:50.594476 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c933115e-f7d6-4878-8d5f-a4c81447652d-istio-token\") pod \"maas-default-gateway-openshift-default-5ccb968c66-hc5bh\" (UID: \"c933115e-f7d6-4878-8d5f-a4c81447652d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-hc5bh" Sep 10 13:24:50.594514 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:24:50.594499 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c933115e-f7d6-4878-8d5f-a4c81447652d-istio-data\") pod \"maas-default-gateway-openshift-default-5ccb968c66-hc5bh\" (UID: \"c933115e-f7d6-4878-8d5f-a4c81447652d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-hc5bh" Sep 10 13:24:50.594777 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:24:50.594550 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c933115e-f7d6-4878-8d5f-a4c81447652d-istio-envoy\") pod \"maas-default-gateway-openshift-default-5ccb968c66-hc5bh\" (UID: \"c933115e-f7d6-4878-8d5f-a4c81447652d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-hc5bh" Sep 10 13:24:50.594777 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:24:50.594573 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c933115e-f7d6-4878-8d5f-a4c81447652d-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-5ccb968c66-hc5bh\" (UID: \"c933115e-f7d6-4878-8d5f-a4c81447652d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-hc5bh" Sep 10 13:24:50.594777 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:24:50.594604 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pm5mj\" (UniqueName: \"kubernetes.io/projected/c933115e-f7d6-4878-8d5f-a4c81447652d-kube-api-access-pm5mj\") pod \"maas-default-gateway-openshift-default-5ccb968c66-hc5bh\" (UID: \"c933115e-f7d6-4878-8d5f-a4c81447652d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-hc5bh" Sep 10 13:24:50.594926 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:24:50.594782 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c933115e-f7d6-4878-8d5f-a4c81447652d-workload-certs\") pod \"maas-default-gateway-openshift-default-5ccb968c66-hc5bh\" (UID: \"c933115e-f7d6-4878-8d5f-a4c81447652d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-hc5bh" Sep 10 13:24:50.594926 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:24:50.594825 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c933115e-f7d6-4878-8d5f-a4c81447652d-workload-socket\") pod \"maas-default-gateway-openshift-default-5ccb968c66-hc5bh\" (UID: \"c933115e-f7d6-4878-8d5f-a4c81447652d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-hc5bh" Sep 10 13:24:50.594926 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:24:50.594888 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c933115e-f7d6-4878-8d5f-a4c81447652d-istio-data\") pod \"maas-default-gateway-openshift-default-5ccb968c66-hc5bh\" (UID: \"c933115e-f7d6-4878-8d5f-a4c81447652d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-hc5bh" Sep 10 13:24:50.595107 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:24:50.595088 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c933115e-f7d6-4878-8d5f-a4c81447652d-credential-socket\") pod \"maas-default-gateway-openshift-default-5ccb968c66-hc5bh\" (UID: \"c933115e-f7d6-4878-8d5f-a4c81447652d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-hc5bh" Sep 10 13:24:50.595277 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:24:50.595258 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c933115e-f7d6-4878-8d5f-a4c81447652d-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-5ccb968c66-hc5bh\" (UID: \"c933115e-f7d6-4878-8d5f-a4c81447652d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-hc5bh" Sep 10 13:24:50.597015 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:24:50.596994 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c933115e-f7d6-4878-8d5f-a4c81447652d-istio-podinfo\") pod \"maas-default-gateway-openshift-default-5ccb968c66-hc5bh\" (UID: \"c933115e-f7d6-4878-8d5f-a4c81447652d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-hc5bh" Sep 10 13:24:50.597113 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:24:50.597047 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c933115e-f7d6-4878-8d5f-a4c81447652d-istio-envoy\") pod \"maas-default-gateway-openshift-default-5ccb968c66-hc5bh\" (UID: \"c933115e-f7d6-4878-8d5f-a4c81447652d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-hc5bh" Sep 10 13:24:50.602668 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:24:50.602637 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pm5mj\" (UniqueName: \"kubernetes.io/projected/c933115e-f7d6-4878-8d5f-a4c81447652d-kube-api-access-pm5mj\") pod \"maas-default-gateway-openshift-default-5ccb968c66-hc5bh\" (UID: \"c933115e-f7d6-4878-8d5f-a4c81447652d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-hc5bh" Sep 10 13:24:50.603389 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:24:50.603370 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c933115e-f7d6-4878-8d5f-a4c81447652d-istio-token\") pod \"maas-default-gateway-openshift-default-5ccb968c66-hc5bh\" (UID: \"c933115e-f7d6-4878-8d5f-a4c81447652d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-hc5bh" Sep 10 13:24:50.732465 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:24:50.732405 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-hc5bh" Sep 10 13:24:51.013296 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:24:51.013274 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-hc5bh"] Sep 10 13:24:51.016105 ip-10-0-139-174 kubenswrapper[2557]: W0910 13:24:51.016064 2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc933115e_f7d6_4878_8d5f_a4c81447652d.slice/crio-3ffa1ab086dce681c79b74ba92f2bc5270fd6f158a018a71346f82a07c615e21 WatchSource:0}: Error finding container 3ffa1ab086dce681c79b74ba92f2bc5270fd6f158a018a71346f82a07c615e21: Status 404 returned error can't find the container with id 3ffa1ab086dce681c79b74ba92f2bc5270fd6f158a018a71346f82a07c615e21 Sep 10 13:24:51.018366 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:24:51.018334 2557 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892152Ki","pods":"250"} Sep 10 13:24:51.018455 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:24:51.018404 2557 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892152Ki","pods":"250"} Sep 10 13:24:51.018498 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:24:51.018452 2557 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892152Ki","pods":"250"} Sep 10 13:24:51.421780 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:24:51.421698 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-hc5bh" event={"ID":"c933115e-f7d6-4878-8d5f-a4c81447652d","Type":"ContainerStarted","Data":"ca98dce3324e45e7bb0016d8e025cc076c9945be2a462a98c917b4d1e59c89e6"} Sep 10 13:24:51.421780 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:24:51.421747 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-hc5bh" event={"ID":"c933115e-f7d6-4878-8d5f-a4c81447652d","Type":"ContainerStarted","Data":"3ffa1ab086dce681c79b74ba92f2bc5270fd6f158a018a71346f82a07c615e21"} Sep 10 13:24:51.447323 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:24:51.447261 2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-hc5bh" podStartSLOduration=1.447242533 podStartE2EDuration="1.447242533s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 13:24:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 13:24:51.446282329 +0000 UTC m=+451.753276744" watchObservedRunningTime="2026-09-10 13:24:51.447242533 +0000 UTC m=+451.754236948" Sep 10 13:24:51.732958 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:24:51.732927 2557 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-hc5bh" Sep 10 13:24:52.737952 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:24:52.737923 2557 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-hc5bh" Sep 10 13:24:53.432492 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:24:53.432460 2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-hc5bh" Sep 10 13:24:53.433300 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:24:53.433279 2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-hc5bh" Sep 10 13:25:45.833876 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:25:45.833843 2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-7878d84c4d-hvwcv"] Sep 10 13:25:45.837028 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:25:45.837014 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-7878d84c4d-hvwcv" Sep 10 13:25:45.839872 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:25:45.839841 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\"" Sep 10 13:25:45.840009 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:25:45.839888 2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\"" Sep 10 13:25:45.840151 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:25:45.840136 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-jc6jm\"" Sep 10 13:25:45.840596 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:25:45.840573 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-metrics-tls\"" Sep 10 13:25:45.848967 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:25:45.848943 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-7878d84c4d-hvwcv"] Sep 10 13:25:45.935098 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:25:45.935071 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/575f105f-5487-444e-8690-0eba38df97c3-webhook-cert\") pod \"maas-controller-7878d84c4d-hvwcv\" (UID: \"575f105f-5487-444e-8690-0eba38df97c3\") " pod="opendatahub/maas-controller-7878d84c4d-hvwcv" Sep 10 13:25:45.935225 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:25:45.935122 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-495rf\" (UniqueName: \"kubernetes.io/projected/575f105f-5487-444e-8690-0eba38df97c3-kube-api-access-495rf\") pod \"maas-controller-7878d84c4d-hvwcv\" (UID: \"575f105f-5487-444e-8690-0eba38df97c3\") " pod="opendatahub/maas-controller-7878d84c4d-hvwcv" Sep 10 13:25:45.935225 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:25:45.935180 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/575f105f-5487-444e-8690-0eba38df97c3-metrics-certs\") pod \"maas-controller-7878d84c4d-hvwcv\" (UID: \"575f105f-5487-444e-8690-0eba38df97c3\") " pod="opendatahub/maas-controller-7878d84c4d-hvwcv" Sep 10 13:25:46.036540 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:25:46.036512 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-495rf\" (UniqueName: \"kubernetes.io/projected/575f105f-5487-444e-8690-0eba38df97c3-kube-api-access-495rf\") pod \"maas-controller-7878d84c4d-hvwcv\" (UID: \"575f105f-5487-444e-8690-0eba38df97c3\") " pod="opendatahub/maas-controller-7878d84c4d-hvwcv" Sep 10 13:25:46.036649 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:25:46.036560 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/575f105f-5487-444e-8690-0eba38df97c3-metrics-certs\") pod \"maas-controller-7878d84c4d-hvwcv\" (UID: \"575f105f-5487-444e-8690-0eba38df97c3\") " pod="opendatahub/maas-controller-7878d84c4d-hvwcv" Sep 10 13:25:46.036649 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:25:46.036644 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/575f105f-5487-444e-8690-0eba38df97c3-webhook-cert\") pod \"maas-controller-7878d84c4d-hvwcv\" (UID: \"575f105f-5487-444e-8690-0eba38df97c3\") " pod="opendatahub/maas-controller-7878d84c4d-hvwcv" Sep 10 13:25:46.039186 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:25:46.039155 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/575f105f-5487-444e-8690-0eba38df97c3-metrics-certs\") pod \"maas-controller-7878d84c4d-hvwcv\" (UID: \"575f105f-5487-444e-8690-0eba38df97c3\") " pod="opendatahub/maas-controller-7878d84c4d-hvwcv" Sep 10 13:25:46.039288 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:25:46.039197 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/575f105f-5487-444e-8690-0eba38df97c3-webhook-cert\") pod \"maas-controller-7878d84c4d-hvwcv\" (UID: \"575f105f-5487-444e-8690-0eba38df97c3\") " pod="opendatahub/maas-controller-7878d84c4d-hvwcv" Sep 10 13:25:46.045795 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:25:46.045773 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-495rf\" (UniqueName: \"kubernetes.io/projected/575f105f-5487-444e-8690-0eba38df97c3-kube-api-access-495rf\") pod \"maas-controller-7878d84c4d-hvwcv\" (UID: \"575f105f-5487-444e-8690-0eba38df97c3\") " pod="opendatahub/maas-controller-7878d84c4d-hvwcv" Sep 10 13:25:46.148758 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:25:46.148689 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-7878d84c4d-hvwcv" Sep 10 13:25:46.432883 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:25:46.432859 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-7878d84c4d-hvwcv"] Sep 10 13:25:46.434646 ip-10-0-139-174 kubenswrapper[2557]: W0910 13:25:46.434619 2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod575f105f_5487_444e_8690_0eba38df97c3.slice/crio-324042d4a04665d1a2d4c5fa9f88d331e803bd534d31c6a6e5313d35316015da WatchSource:0}: Error finding container 324042d4a04665d1a2d4c5fa9f88d331e803bd534d31c6a6e5313d35316015da: Status 404 returned error can't find the container with id 324042d4a04665d1a2d4c5fa9f88d331e803bd534d31c6a6e5313d35316015da Sep 10 13:25:46.702669 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:25:46.702635 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-7878d84c4d-hvwcv" event={"ID":"575f105f-5487-444e-8690-0eba38df97c3","Type":"ContainerStarted","Data":"324042d4a04665d1a2d4c5fa9f88d331e803bd534d31c6a6e5313d35316015da"} Sep 10 13:25:49.725240 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:25:49.725201 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-7878d84c4d-hvwcv" event={"ID":"575f105f-5487-444e-8690-0eba38df97c3","Type":"ContainerStarted","Data":"4a54dab47090b536b25a4f5a6fc1de7822fbeed6c9e44e6521f64b09e97cfcb7"} Sep 10 13:25:49.725661 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:25:49.725272 2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-7878d84c4d-hvwcv" Sep 10 13:25:49.749717 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:25:49.749659 2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-controller-7878d84c4d-hvwcv" podStartSLOduration=2.39072659 podStartE2EDuration="4.749640965s" totalImagesPullingTime="2.358914375s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 13:25:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 13:25:49.747798789 +0000 UTC m=+510.054793193" watchObservedRunningTime="2026-09-10 13:25:49.749640965 +0000 UTC m=+510.056635380" Sep 10 13:26:00.733228 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:26:00.733198 2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-7878d84c4d-hvwcv" Sep 10 13:26:22.814381 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:26:22.814339 2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-5b54b7fbb7-t2dzv"] Sep 10 13:26:22.819938 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:26:22.819914 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-5b54b7fbb7-t2dzv" Sep 10 13:26:22.822313 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:26:22.822285 2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins\"" Sep 10 13:26:22.822508 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:26:22.822412 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-dockercfg-5bw6l\"" Sep 10 13:26:22.832533 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:26:22.832508 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-5b54b7fbb7-t2dzv"] Sep 10 13:26:22.940535 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:26:22.940505 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9kg2p\" (UniqueName: \"kubernetes.io/projected/68b5cdbd-ff82-4dc9-aa29-c592a12629e0-kube-api-access-9kg2p\") pod \"payload-processing-5b54b7fbb7-t2dzv\" (UID: \"68b5cdbd-ff82-4dc9-aa29-c592a12629e0\") " pod="openshift-ingress/payload-processing-5b54b7fbb7-t2dzv" Sep 10 13:26:22.940701 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:26:22.940631 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/68b5cdbd-ff82-4dc9-aa29-c592a12629e0-plugins-config-volume\") pod \"payload-processing-5b54b7fbb7-t2dzv\" (UID: \"68b5cdbd-ff82-4dc9-aa29-c592a12629e0\") " pod="openshift-ingress/payload-processing-5b54b7fbb7-t2dzv" Sep 10 13:26:23.041406 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:26:23.041373 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9kg2p\" (UniqueName: \"kubernetes.io/projected/68b5cdbd-ff82-4dc9-aa29-c592a12629e0-kube-api-access-9kg2p\") pod \"payload-processing-5b54b7fbb7-t2dzv\" (UID: \"68b5cdbd-ff82-4dc9-aa29-c592a12629e0\") " pod="openshift-ingress/payload-processing-5b54b7fbb7-t2dzv" Sep 10 13:26:23.041605 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:26:23.041482 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/68b5cdbd-ff82-4dc9-aa29-c592a12629e0-plugins-config-volume\") pod \"payload-processing-5b54b7fbb7-t2dzv\" (UID: \"68b5cdbd-ff82-4dc9-aa29-c592a12629e0\") " pod="openshift-ingress/payload-processing-5b54b7fbb7-t2dzv" Sep 10 13:26:23.042114 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:26:23.042096 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/68b5cdbd-ff82-4dc9-aa29-c592a12629e0-plugins-config-volume\") pod \"payload-processing-5b54b7fbb7-t2dzv\" (UID: \"68b5cdbd-ff82-4dc9-aa29-c592a12629e0\") " pod="openshift-ingress/payload-processing-5b54b7fbb7-t2dzv" Sep 10 13:26:23.050616 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:26:23.050590 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9kg2p\" (UniqueName: \"kubernetes.io/projected/68b5cdbd-ff82-4dc9-aa29-c592a12629e0-kube-api-access-9kg2p\") pod \"payload-processing-5b54b7fbb7-t2dzv\" (UID: \"68b5cdbd-ff82-4dc9-aa29-c592a12629e0\") " pod="openshift-ingress/payload-processing-5b54b7fbb7-t2dzv" Sep 10 13:26:23.130526 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:26:23.130460 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-5b54b7fbb7-t2dzv" Sep 10 13:26:23.409476 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:26:23.409452 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-5b54b7fbb7-t2dzv"] Sep 10 13:26:23.411934 ip-10-0-139-174 kubenswrapper[2557]: W0910 13:26:23.411909 2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod68b5cdbd_ff82_4dc9_aa29_c592a12629e0.slice/crio-a3ef123a68255486ccf83145764ccf7b7b63a330529c7b48e74dc864cf7aa1ec WatchSource:0}: Error finding container a3ef123a68255486ccf83145764ccf7b7b63a330529c7b48e74dc864cf7aa1ec: Status 404 returned error can't find the container with id a3ef123a68255486ccf83145764ccf7b7b63a330529c7b48e74dc864cf7aa1ec Sep 10 13:26:23.902037 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:26:23.902009 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-5b54b7fbb7-t2dzv" event={"ID":"68b5cdbd-ff82-4dc9-aa29-c592a12629e0","Type":"ContainerStarted","Data":"a3ef123a68255486ccf83145764ccf7b7b63a330529c7b48e74dc864cf7aa1ec"} Sep 10 13:26:25.916024 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:26:25.915927 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-5b54b7fbb7-t2dzv" event={"ID":"68b5cdbd-ff82-4dc9-aa29-c592a12629e0","Type":"ContainerStarted","Data":"c78e64f944286aacef661827b5b1ab456f2d67c6a3f9fcba61867c5d2011dc83"} Sep 10 13:26:25.916024 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:26:25.916017 2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-5b54b7fbb7-t2dzv" Sep 10 13:26:25.940049 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:26:25.939810 2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-5b54b7fbb7-t2dzv" podStartSLOduration=1.785492388 podStartE2EDuration="3.939792135s" totalImagesPullingTime="2.154299747s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 13:26:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 13:26:25.939060874 +0000 UTC m=+546.246055288" watchObservedRunningTime="2026-09-10 13:26:25.939792135 +0000 UTC m=+546.246786563" Sep 10 13:26:32.954511 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:26:32.954480 2557 generic.go:356] "Generic (PLEG): container finished" podID="c933115e-f7d6-4878-8d5f-a4c81447652d" containerID="ca98dce3324e45e7bb0016d8e025cc076c9945be2a462a98c917b4d1e59c89e6" exitCode=0 Sep 10 13:26:32.954909 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:26:32.954565 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-hc5bh" event={"ID":"c933115e-f7d6-4878-8d5f-a4c81447652d","Type":"ContainerDied","Data":"ca98dce3324e45e7bb0016d8e025cc076c9945be2a462a98c917b4d1e59c89e6"} Sep 10 13:26:32.954955 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:26:32.954926 2557 scope.go:117] "RemoveContainer" containerID="ca98dce3324e45e7bb0016d8e025cc076c9945be2a462a98c917b4d1e59c89e6" Sep 10 13:26:32.955881 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:26:32.955853 2557 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892152Ki","pods":"250"} Sep 10 13:26:32.955954 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:26:32.955938 2557 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892152Ki","pods":"250"} Sep 10 13:26:32.956006 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:26:32.955992 2557 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892152Ki","pods":"250"} Sep 10 13:26:33.966862 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:26:33.966829 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-hc5bh" event={"ID":"c933115e-f7d6-4878-8d5f-a4c81447652d","Type":"ContainerStarted","Data":"a7c4182b0d47aec1d9b54bac9e0d6bdc2118eb955a439bf11adb54b5ec8601d7"} Sep 10 13:26:34.733301 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:26:34.733267 2557 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-hc5bh" Sep 10 13:26:34.734222 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:26:34.734193 2557 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-5ccb968c66-hc5bh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.37:15021/healthz/ready\": dial tcp 10.134.0.37:15021: connect: connection refused" start-of-body= Sep 10 13:26:34.734372 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:26:34.734240 2557 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-hc5bh" podUID="c933115e-f7d6-4878-8d5f-a4c81447652d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.37:15021/healthz/ready\": dial tcp 10.134.0.37:15021: connect: connection refused" Sep 10 13:26:35.733110 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:26:35.733075 2557 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-5ccb968c66-hc5bh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.37:15021/healthz/ready\": dial tcp 10.134.0.37:15021: connect: connection refused" start-of-body= Sep 10 13:26:35.733518 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:26:35.733140 2557 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-hc5bh" podUID="c933115e-f7d6-4878-8d5f-a4c81447652d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.37:15021/healthz/ready\": dial tcp 10.134.0.37:15021: connect: connection refused" Sep 10 13:26:36.733700 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:26:36.733666 2557 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-5ccb968c66-hc5bh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.37:15021/healthz/ready\": dial tcp 10.134.0.37:15021: connect: connection refused" start-of-body= Sep 10 13:26:36.734087 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:26:36.733720 2557 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-hc5bh" podUID="c933115e-f7d6-4878-8d5f-a4c81447652d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.37:15021/healthz/ready\": dial tcp 10.134.0.37:15021: connect: connection refused" Sep 10 13:26:36.923971 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:26:36.923935 2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-5b54b7fbb7-t2dzv" Sep 10 13:26:37.733617 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:26:37.733585 2557 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-5ccb968c66-hc5bh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.37:15021/healthz/ready\": dial tcp 10.134.0.37:15021: connect: connection refused" start-of-body= Sep 10 13:26:37.733791 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:26:37.733636 2557 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-hc5bh" podUID="c933115e-f7d6-4878-8d5f-a4c81447652d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.37:15021/healthz/ready\": dial tcp 10.134.0.37:15021: connect: connection refused" Sep 10 13:26:38.433298 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:26:38.433260 2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-hc5bh" Sep 10 13:26:38.733365 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:26:38.733327 2557 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-5ccb968c66-hc5bh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.37:15021/healthz/ready\": dial tcp 10.134.0.37:15021: connect: connection refused" start-of-body= Sep 10 13:26:38.733558 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:26:38.733389 2557 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-hc5bh" podUID="c933115e-f7d6-4878-8d5f-a4c81447652d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.37:15021/healthz/ready\": dial tcp 10.134.0.37:15021: connect: connection refused" Sep 10 13:26:38.930631 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:26:38.930597 2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-jg757"] Sep 10 13:26:38.933683 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:26:38.933668 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-jg757" Sep 10 13:26:38.936392 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:26:38.936348 2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\"" Sep 10 13:26:38.936392 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:26:38.936376 2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\"" Sep 10 13:26:38.936392 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:26:38.936392 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-g6wbm\"" Sep 10 13:26:38.936392 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:26:38.936402 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-distinct-2-simulated-kserve-self-signed-certs\"" Sep 10 13:26:38.945391 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:26:38.945369 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-jg757"] Sep 10 13:26:39.005125 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:26:39.005041 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a8295d66-90d3-4610-adfe-1e6c03c04a0d-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-jg757\" (UID: \"a8295d66-90d3-4610-adfe-1e6c03c04a0d\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-jg757" Sep 10 13:26:39.005125 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:26:39.005093 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a8295d66-90d3-4610-adfe-1e6c03c04a0d-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-jg757\" (UID: \"a8295d66-90d3-4610-adfe-1e6c03c04a0d\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-jg757" Sep 10 13:26:39.005325 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:26:39.005184 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a8295d66-90d3-4610-adfe-1e6c03c04a0d-model-cache\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-jg757\" (UID: \"a8295d66-90d3-4610-adfe-1e6c03c04a0d\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-jg757" Sep 10 13:26:39.005325 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:26:39.005219 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a8295d66-90d3-4610-adfe-1e6c03c04a0d-dshm\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-jg757\" (UID: \"a8295d66-90d3-4610-adfe-1e6c03c04a0d\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-jg757" Sep 10 13:26:39.005325 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:26:39.005241 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jcjgp\" (UniqueName: \"kubernetes.io/projected/a8295d66-90d3-4610-adfe-1e6c03c04a0d-kube-api-access-jcjgp\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-jg757\" (UID: \"a8295d66-90d3-4610-adfe-1e6c03c04a0d\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-jg757" Sep 10 13:26:39.005483 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:26:39.005322 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a8295d66-90d3-4610-adfe-1e6c03c04a0d-home\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-jg757\" (UID: \"a8295d66-90d3-4610-adfe-1e6c03c04a0d\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-jg757" Sep 10 13:26:39.105974 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:26:39.105939 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a8295d66-90d3-4610-adfe-1e6c03c04a0d-model-cache\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-jg757\" (UID: \"a8295d66-90d3-4610-adfe-1e6c03c04a0d\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-jg757" Sep 10 13:26:39.105974 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:26:39.105986 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a8295d66-90d3-4610-adfe-1e6c03c04a0d-dshm\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-jg757\" (UID: \"a8295d66-90d3-4610-adfe-1e6c03c04a0d\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-jg757" Sep 10 13:26:39.106208 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:26:39.106009 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jcjgp\" (UniqueName: \"kubernetes.io/projected/a8295d66-90d3-4610-adfe-1e6c03c04a0d-kube-api-access-jcjgp\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-jg757\" (UID: \"a8295d66-90d3-4610-adfe-1e6c03c04a0d\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-jg757" Sep 10 13:26:39.106208 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:26:39.106069 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a8295d66-90d3-4610-adfe-1e6c03c04a0d-home\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-jg757\" (UID: \"a8295d66-90d3-4610-adfe-1e6c03c04a0d\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-jg757" Sep 10 13:26:39.106208 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:26:39.106109 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a8295d66-90d3-4610-adfe-1e6c03c04a0d-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-jg757\" (UID: \"a8295d66-90d3-4610-adfe-1e6c03c04a0d\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-jg757" Sep 10 13:26:39.106208 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:26:39.106150 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a8295d66-90d3-4610-adfe-1e6c03c04a0d-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-jg757\" (UID: \"a8295d66-90d3-4610-adfe-1e6c03c04a0d\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-jg757" Sep 10 13:26:39.106409 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:26:39.106388 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a8295d66-90d3-4610-adfe-1e6c03c04a0d-model-cache\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-jg757\" (UID: \"a8295d66-90d3-4610-adfe-1e6c03c04a0d\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-jg757" Sep 10 13:26:39.106541 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:26:39.106521 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a8295d66-90d3-4610-adfe-1e6c03c04a0d-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-jg757\" (UID: \"a8295d66-90d3-4610-adfe-1e6c03c04a0d\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-jg757" Sep 10 13:26:39.106591 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:26:39.106565 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a8295d66-90d3-4610-adfe-1e6c03c04a0d-home\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-jg757\" (UID: \"a8295d66-90d3-4610-adfe-1e6c03c04a0d\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-jg757" Sep 10 13:26:39.108371 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:26:39.108348 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a8295d66-90d3-4610-adfe-1e6c03c04a0d-dshm\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-jg757\" (UID: \"a8295d66-90d3-4610-adfe-1e6c03c04a0d\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-jg757" Sep 10 13:26:39.108707 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:26:39.108689 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a8295d66-90d3-4610-adfe-1e6c03c04a0d-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-jg757\" (UID: \"a8295d66-90d3-4610-adfe-1e6c03c04a0d\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-jg757" Sep 10 13:26:39.115290 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:26:39.115269 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jcjgp\" (UniqueName: \"kubernetes.io/projected/a8295d66-90d3-4610-adfe-1e6c03c04a0d-kube-api-access-jcjgp\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-jg757\" (UID: \"a8295d66-90d3-4610-adfe-1e6c03c04a0d\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-jg757" Sep 10 13:26:39.245276 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:26:39.245231 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-jg757" Sep 10 13:26:39.527527 ip-10-0-139-174 kubenswrapper[2557]: W0910 13:26:39.527501 2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda8295d66_90d3_4610_adfe_1e6c03c04a0d.slice/crio-3da92384cf77e163d60681fa6d1bbc3db723516c9ed13921270ad92d348e3c60 WatchSource:0}: Error finding container 3da92384cf77e163d60681fa6d1bbc3db723516c9ed13921270ad92d348e3c60: Status 404 returned error can't find the container with id 3da92384cf77e163d60681fa6d1bbc3db723516c9ed13921270ad92d348e3c60 Sep 10 13:26:39.528203 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:26:39.528180 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-jg757"] Sep 10 13:26:39.736575 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:26:39.736551 2557 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-hc5bh" Sep 10 13:26:39.737503 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:26:39.737481 2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-hc5bh" Sep 10 13:26:40.001629 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:26:40.001381 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-jg757" event={"ID":"a8295d66-90d3-4610-adfe-1e6c03c04a0d","Type":"ContainerStarted","Data":"3da92384cf77e163d60681fa6d1bbc3db723516c9ed13921270ad92d348e3c60"} Sep 10 13:26:41.008163 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:26:41.008129 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-jg757" event={"ID":"a8295d66-90d3-4610-adfe-1e6c03c04a0d","Type":"ContainerStarted","Data":"6a5087e7022b693fcb263cffb5d60e144c7cc1819c16fbad17d5a5bde65a976d"} Sep 10 13:26:49.245384 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:26:49.245345 2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-jg757" Sep 10 13:26:49.245775 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:26:49.245500 2557 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-jg757" Sep 10 13:26:49.257706 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:26:49.257685 2557 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-jg757" Sep 10 13:26:49.298059 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:26:49.298012 2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-jg757" podStartSLOduration=10.162330744 podStartE2EDuration="11.297998023s" totalImagesPullingTime="1.135667279s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 13:26:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 13:26:41.031275731 +0000 UTC m=+561.338270138" watchObservedRunningTime="2026-09-10 13:26:49.297998023 +0000 UTC m=+569.604992437" Sep 10 13:26:50.064838 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:26:50.064809 2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-jg757" Sep 10 13:26:56.914309 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:26:56.914263 2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-598466548d-q7pvs"] Sep 10 13:26:56.917256 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:26:56.917210 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-q7pvs" Sep 10 13:26:56.919920 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:26:56.919897 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-trlp-test-simulated-kserve-self-signed-certs\"" Sep 10 13:26:56.938132 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:26:56.938108 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-598466548d-q7pvs"] Sep 10 13:26:57.060740 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:26:57.060707 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ae1519ac-d912-4bf6-b1bf-5b0a588899f0-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-598466548d-q7pvs\" (UID: \"ae1519ac-d912-4bf6-b1bf-5b0a588899f0\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-q7pvs" Sep 10 13:26:57.060949 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:26:57.060771 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ae1519ac-d912-4bf6-b1bf-5b0a588899f0-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-598466548d-q7pvs\" (UID: \"ae1519ac-d912-4bf6-b1bf-5b0a588899f0\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-q7pvs" Sep 10 13:26:57.060949 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:26:57.060834 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ae1519ac-d912-4bf6-b1bf-5b0a588899f0-model-cache\") pod \"e2e-trlp-test-simulated-kserve-598466548d-q7pvs\" (UID: \"ae1519ac-d912-4bf6-b1bf-5b0a588899f0\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-q7pvs" Sep 10 13:26:57.060949 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:26:57.060879 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8bzv8\" (UniqueName: \"kubernetes.io/projected/ae1519ac-d912-4bf6-b1bf-5b0a588899f0-kube-api-access-8bzv8\") pod \"e2e-trlp-test-simulated-kserve-598466548d-q7pvs\" (UID: \"ae1519ac-d912-4bf6-b1bf-5b0a588899f0\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-q7pvs" Sep 10 13:26:57.061111 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:26:57.060948 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ae1519ac-d912-4bf6-b1bf-5b0a588899f0-home\") pod \"e2e-trlp-test-simulated-kserve-598466548d-q7pvs\" (UID: \"ae1519ac-d912-4bf6-b1bf-5b0a588899f0\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-q7pvs" Sep 10 13:26:57.061111 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:26:57.060975 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ae1519ac-d912-4bf6-b1bf-5b0a588899f0-dshm\") pod \"e2e-trlp-test-simulated-kserve-598466548d-q7pvs\" (UID: \"ae1519ac-d912-4bf6-b1bf-5b0a588899f0\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-q7pvs" Sep 10 13:26:57.161936 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:26:57.161903 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ae1519ac-d912-4bf6-b1bf-5b0a588899f0-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-598466548d-q7pvs\" (UID: \"ae1519ac-d912-4bf6-b1bf-5b0a588899f0\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-q7pvs" Sep 10 13:26:57.162076 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:26:57.161948 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ae1519ac-d912-4bf6-b1bf-5b0a588899f0-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-598466548d-q7pvs\" (UID: \"ae1519ac-d912-4bf6-b1bf-5b0a588899f0\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-q7pvs" Sep 10 13:26:57.162076 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:26:57.161973 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ae1519ac-d912-4bf6-b1bf-5b0a588899f0-model-cache\") pod \"e2e-trlp-test-simulated-kserve-598466548d-q7pvs\" (UID: \"ae1519ac-d912-4bf6-b1bf-5b0a588899f0\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-q7pvs" Sep 10 13:26:57.162076 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:26:57.162015 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8bzv8\" (UniqueName: \"kubernetes.io/projected/ae1519ac-d912-4bf6-b1bf-5b0a588899f0-kube-api-access-8bzv8\") pod \"e2e-trlp-test-simulated-kserve-598466548d-q7pvs\" (UID: \"ae1519ac-d912-4bf6-b1bf-5b0a588899f0\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-q7pvs" Sep 10 13:26:57.162076 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:26:57.162063 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ae1519ac-d912-4bf6-b1bf-5b0a588899f0-home\") pod \"e2e-trlp-test-simulated-kserve-598466548d-q7pvs\" (UID: \"ae1519ac-d912-4bf6-b1bf-5b0a588899f0\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-q7pvs" Sep 10 13:26:57.162284 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:26:57.162091 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ae1519ac-d912-4bf6-b1bf-5b0a588899f0-dshm\") pod \"e2e-trlp-test-simulated-kserve-598466548d-q7pvs\" (UID: \"ae1519ac-d912-4bf6-b1bf-5b0a588899f0\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-q7pvs" Sep 10 13:26:57.162358 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:26:57.162339 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ae1519ac-d912-4bf6-b1bf-5b0a588899f0-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-598466548d-q7pvs\" (UID: \"ae1519ac-d912-4bf6-b1bf-5b0a588899f0\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-q7pvs" Sep 10 13:26:57.162448 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:26:57.162381 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ae1519ac-d912-4bf6-b1bf-5b0a588899f0-model-cache\") pod \"e2e-trlp-test-simulated-kserve-598466548d-q7pvs\" (UID: \"ae1519ac-d912-4bf6-b1bf-5b0a588899f0\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-q7pvs" Sep 10 13:26:57.162515 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:26:57.162477 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ae1519ac-d912-4bf6-b1bf-5b0a588899f0-home\") pod \"e2e-trlp-test-simulated-kserve-598466548d-q7pvs\" (UID: \"ae1519ac-d912-4bf6-b1bf-5b0a588899f0\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-q7pvs" Sep 10 13:26:57.164443 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:26:57.164386 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ae1519ac-d912-4bf6-b1bf-5b0a588899f0-dshm\") pod \"e2e-trlp-test-simulated-kserve-598466548d-q7pvs\" (UID: \"ae1519ac-d912-4bf6-b1bf-5b0a588899f0\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-q7pvs" Sep 10 13:26:57.164534 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:26:57.164517 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ae1519ac-d912-4bf6-b1bf-5b0a588899f0-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-598466548d-q7pvs\" (UID: \"ae1519ac-d912-4bf6-b1bf-5b0a588899f0\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-q7pvs" Sep 10 13:26:57.170725 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:26:57.170703 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8bzv8\" (UniqueName: \"kubernetes.io/projected/ae1519ac-d912-4bf6-b1bf-5b0a588899f0-kube-api-access-8bzv8\") pod \"e2e-trlp-test-simulated-kserve-598466548d-q7pvs\" (UID: \"ae1519ac-d912-4bf6-b1bf-5b0a588899f0\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-q7pvs" Sep 10 13:26:57.227301 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:26:57.227264 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-q7pvs" Sep 10 13:26:57.500872 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:26:57.500846 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-598466548d-q7pvs"] Sep 10 13:26:57.502268 ip-10-0-139-174 kubenswrapper[2557]: W0910 13:26:57.502240 2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podae1519ac_d912_4bf6_b1bf_5b0a588899f0.slice/crio-e889edf38fdd635c028f138112af4b43ef0bee8e66586b308dbdc7339ae3bcac WatchSource:0}: Error finding container e889edf38fdd635c028f138112af4b43ef0bee8e66586b308dbdc7339ae3bcac: Status 404 returned error can't find the container with id e889edf38fdd635c028f138112af4b43ef0bee8e66586b308dbdc7339ae3bcac Sep 10 13:26:58.098709 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:26:58.098679 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-q7pvs" event={"ID":"ae1519ac-d912-4bf6-b1bf-5b0a588899f0","Type":"ContainerStarted","Data":"681ff14af5e17c1efc0e315a6c994a1cbcd94d45cc00b5f956eab57dff8f7930"} Sep 10 13:26:58.099077 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:26:58.098726 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-q7pvs" event={"ID":"ae1519ac-d912-4bf6-b1bf-5b0a588899f0","Type":"ContainerStarted","Data":"e889edf38fdd635c028f138112af4b43ef0bee8e66586b308dbdc7339ae3bcac"} Sep 10 13:26:58.124924 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:26:58.124877 2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-q7pvs" podStartSLOduration=1.9350418820000002 podStartE2EDuration="2.124862203s" totalImagesPullingTime="189.820321ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 13:26:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 13:26:58.122931234 +0000 UTC m=+578.429925651" watchObservedRunningTime="2026-09-10 13:26:58.124862203 +0000 UTC m=+578.431856618" Sep 10 13:27:07.228357 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:27:07.228323 2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-q7pvs" Sep 10 13:27:07.228811 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:27:07.228396 2557 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-q7pvs" Sep 10 13:27:07.240794 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:27:07.240771 2557 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-q7pvs" Sep 10 13:27:08.161559 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:27:08.161530 2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-q7pvs" Sep 10 13:27:20.137561 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:27:20.137526 2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-92n4q_720acda6-0b15-49ac-ab4e-60de70c4324a/ovn-acl-logging/0.log" Sep 10 13:27:20.138590 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:27:20.138570 2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-92n4q_720acda6-0b15-49ac-ab4e-60de70c4324a/ovn-acl-logging/0.log" Sep 10 13:29:22.271900 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:29:22.271859 2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-39df6e-openshift-default-69b8d58b68-cpt89"] Sep 10 13:29:22.275803 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:29:22.275778 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-39df6e-openshift-default-69b8d58b68-cpt89" Sep 10 13:29:22.278084 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:29:22.278057 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-a-39df6e-openshift-default-dockercfg-bc4ql\"" Sep 10 13:29:22.290072 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:29:22.290035 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-39df6e-openshift-default-69b8d58b68-cpt89"] Sep 10 13:29:22.389879 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:29:22.389777 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bbab6794-a024-4b72-a8f2-2b9a4e85a3f2-credential-socket\") pod \"e2e-inf-a-39df6e-openshift-default-69b8d58b68-cpt89\" (UID: \"bbab6794-a024-4b72-a8f2-2b9a4e85a3f2\") " pod="openshift-ingress/e2e-inf-a-39df6e-openshift-default-69b8d58b68-cpt89" Sep 10 13:29:22.389879 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:29:22.389817 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bbab6794-a024-4b72-a8f2-2b9a4e85a3f2-workload-socket\") pod \"e2e-inf-a-39df6e-openshift-default-69b8d58b68-cpt89\" (UID: \"bbab6794-a024-4b72-a8f2-2b9a4e85a3f2\") " pod="openshift-ingress/e2e-inf-a-39df6e-openshift-default-69b8d58b68-cpt89" Sep 10 13:29:22.389879 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:29:22.389848 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bbab6794-a024-4b72-a8f2-2b9a4e85a3f2-istiod-ca-cert\") pod \"e2e-inf-a-39df6e-openshift-default-69b8d58b68-cpt89\" (UID: \"bbab6794-a024-4b72-a8f2-2b9a4e85a3f2\") " pod="openshift-ingress/e2e-inf-a-39df6e-openshift-default-69b8d58b68-cpt89" Sep 10 13:29:22.390166 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:29:22.389939 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zn9z4\" (UniqueName: \"kubernetes.io/projected/bbab6794-a024-4b72-a8f2-2b9a4e85a3f2-kube-api-access-zn9z4\") pod \"e2e-inf-a-39df6e-openshift-default-69b8d58b68-cpt89\" (UID: \"bbab6794-a024-4b72-a8f2-2b9a4e85a3f2\") " pod="openshift-ingress/e2e-inf-a-39df6e-openshift-default-69b8d58b68-cpt89" Sep 10 13:29:22.390166 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:29:22.390033 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bbab6794-a024-4b72-a8f2-2b9a4e85a3f2-istio-podinfo\") pod \"e2e-inf-a-39df6e-openshift-default-69b8d58b68-cpt89\" (UID: \"bbab6794-a024-4b72-a8f2-2b9a4e85a3f2\") " pod="openshift-ingress/e2e-inf-a-39df6e-openshift-default-69b8d58b68-cpt89" Sep 10 13:29:22.390166 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:29:22.390072 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bbab6794-a024-4b72-a8f2-2b9a4e85a3f2-istio-envoy\") pod \"e2e-inf-a-39df6e-openshift-default-69b8d58b68-cpt89\" (UID: \"bbab6794-a024-4b72-a8f2-2b9a4e85a3f2\") " pod="openshift-ingress/e2e-inf-a-39df6e-openshift-default-69b8d58b68-cpt89" Sep 10 13:29:22.390166 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:29:22.390120 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bbab6794-a024-4b72-a8f2-2b9a4e85a3f2-workload-certs\") pod \"e2e-inf-a-39df6e-openshift-default-69b8d58b68-cpt89\" (UID: \"bbab6794-a024-4b72-a8f2-2b9a4e85a3f2\") " pod="openshift-ingress/e2e-inf-a-39df6e-openshift-default-69b8d58b68-cpt89" Sep 10 13:29:22.390337 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:29:22.390186 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bbab6794-a024-4b72-a8f2-2b9a4e85a3f2-istio-token\") pod \"e2e-inf-a-39df6e-openshift-default-69b8d58b68-cpt89\" (UID: \"bbab6794-a024-4b72-a8f2-2b9a4e85a3f2\") " pod="openshift-ingress/e2e-inf-a-39df6e-openshift-default-69b8d58b68-cpt89" Sep 10 13:29:22.390337 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:29:22.390260 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bbab6794-a024-4b72-a8f2-2b9a4e85a3f2-istio-data\") pod \"e2e-inf-a-39df6e-openshift-default-69b8d58b68-cpt89\" (UID: \"bbab6794-a024-4b72-a8f2-2b9a4e85a3f2\") " pod="openshift-ingress/e2e-inf-a-39df6e-openshift-default-69b8d58b68-cpt89" Sep 10 13:29:22.491658 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:29:22.491613 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bbab6794-a024-4b72-a8f2-2b9a4e85a3f2-istio-envoy\") pod \"e2e-inf-a-39df6e-openshift-default-69b8d58b68-cpt89\" (UID: \"bbab6794-a024-4b72-a8f2-2b9a4e85a3f2\") " pod="openshift-ingress/e2e-inf-a-39df6e-openshift-default-69b8d58b68-cpt89" Sep 10 13:29:22.491895 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:29:22.491679 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bbab6794-a024-4b72-a8f2-2b9a4e85a3f2-workload-certs\") pod \"e2e-inf-a-39df6e-openshift-default-69b8d58b68-cpt89\" (UID: \"bbab6794-a024-4b72-a8f2-2b9a4e85a3f2\") " pod="openshift-ingress/e2e-inf-a-39df6e-openshift-default-69b8d58b68-cpt89" Sep 10 13:29:22.491895 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:29:22.491697 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bbab6794-a024-4b72-a8f2-2b9a4e85a3f2-istio-token\") pod \"e2e-inf-a-39df6e-openshift-default-69b8d58b68-cpt89\" (UID: \"bbab6794-a024-4b72-a8f2-2b9a4e85a3f2\") " pod="openshift-ingress/e2e-inf-a-39df6e-openshift-default-69b8d58b68-cpt89" Sep 10 13:29:22.491895 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:29:22.491716 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bbab6794-a024-4b72-a8f2-2b9a4e85a3f2-istio-data\") pod \"e2e-inf-a-39df6e-openshift-default-69b8d58b68-cpt89\" (UID: \"bbab6794-a024-4b72-a8f2-2b9a4e85a3f2\") " pod="openshift-ingress/e2e-inf-a-39df6e-openshift-default-69b8d58b68-cpt89" Sep 10 13:29:22.492099 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:29:22.491898 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bbab6794-a024-4b72-a8f2-2b9a4e85a3f2-credential-socket\") pod \"e2e-inf-a-39df6e-openshift-default-69b8d58b68-cpt89\" (UID: \"bbab6794-a024-4b72-a8f2-2b9a4e85a3f2\") " pod="openshift-ingress/e2e-inf-a-39df6e-openshift-default-69b8d58b68-cpt89" Sep 10 13:29:22.492099 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:29:22.491946 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bbab6794-a024-4b72-a8f2-2b9a4e85a3f2-workload-socket\") pod \"e2e-inf-a-39df6e-openshift-default-69b8d58b68-cpt89\" (UID: \"bbab6794-a024-4b72-a8f2-2b9a4e85a3f2\") " pod="openshift-ingress/e2e-inf-a-39df6e-openshift-default-69b8d58b68-cpt89" Sep 10 13:29:22.492099 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:29:22.491988 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bbab6794-a024-4b72-a8f2-2b9a4e85a3f2-istiod-ca-cert\") pod \"e2e-inf-a-39df6e-openshift-default-69b8d58b68-cpt89\" (UID: \"bbab6794-a024-4b72-a8f2-2b9a4e85a3f2\") " pod="openshift-ingress/e2e-inf-a-39df6e-openshift-default-69b8d58b68-cpt89" Sep 10 13:29:22.492099 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:29:22.492019 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zn9z4\" (UniqueName: \"kubernetes.io/projected/bbab6794-a024-4b72-a8f2-2b9a4e85a3f2-kube-api-access-zn9z4\") pod \"e2e-inf-a-39df6e-openshift-default-69b8d58b68-cpt89\" (UID: \"bbab6794-a024-4b72-a8f2-2b9a4e85a3f2\") " pod="openshift-ingress/e2e-inf-a-39df6e-openshift-default-69b8d58b68-cpt89" Sep 10 13:29:22.492099 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:29:22.492067 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bbab6794-a024-4b72-a8f2-2b9a4e85a3f2-istio-podinfo\") pod \"e2e-inf-a-39df6e-openshift-default-69b8d58b68-cpt89\" (UID: \"bbab6794-a024-4b72-a8f2-2b9a4e85a3f2\") " pod="openshift-ingress/e2e-inf-a-39df6e-openshift-default-69b8d58b68-cpt89" Sep 10 13:29:22.492349 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:29:22.492184 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bbab6794-a024-4b72-a8f2-2b9a4e85a3f2-istio-data\") pod \"e2e-inf-a-39df6e-openshift-default-69b8d58b68-cpt89\" (UID: \"bbab6794-a024-4b72-a8f2-2b9a4e85a3f2\") " pod="openshift-ingress/e2e-inf-a-39df6e-openshift-default-69b8d58b68-cpt89" Sep 10 13:29:22.492349 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:29:22.492322 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bbab6794-a024-4b72-a8f2-2b9a4e85a3f2-credential-socket\") pod \"e2e-inf-a-39df6e-openshift-default-69b8d58b68-cpt89\" (UID: \"bbab6794-a024-4b72-a8f2-2b9a4e85a3f2\") " pod="openshift-ingress/e2e-inf-a-39df6e-openshift-default-69b8d58b68-cpt89" Sep 10 13:29:22.492486 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:29:22.492402 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bbab6794-a024-4b72-a8f2-2b9a4e85a3f2-workload-socket\") pod \"e2e-inf-a-39df6e-openshift-default-69b8d58b68-cpt89\" (UID: \"bbab6794-a024-4b72-a8f2-2b9a4e85a3f2\") " pod="openshift-ingress/e2e-inf-a-39df6e-openshift-default-69b8d58b68-cpt89" Sep 10 13:29:22.492486 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:29:22.492479 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bbab6794-a024-4b72-a8f2-2b9a4e85a3f2-workload-certs\") pod \"e2e-inf-a-39df6e-openshift-default-69b8d58b68-cpt89\" (UID: \"bbab6794-a024-4b72-a8f2-2b9a4e85a3f2\") " pod="openshift-ingress/e2e-inf-a-39df6e-openshift-default-69b8d58b68-cpt89" Sep 10 13:29:22.492979 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:29:22.492955 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bbab6794-a024-4b72-a8f2-2b9a4e85a3f2-istiod-ca-cert\") pod \"e2e-inf-a-39df6e-openshift-default-69b8d58b68-cpt89\" (UID: \"bbab6794-a024-4b72-a8f2-2b9a4e85a3f2\") " pod="openshift-ingress/e2e-inf-a-39df6e-openshift-default-69b8d58b68-cpt89" Sep 10 13:29:22.494309 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:29:22.494290 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bbab6794-a024-4b72-a8f2-2b9a4e85a3f2-istio-envoy\") pod \"e2e-inf-a-39df6e-openshift-default-69b8d58b68-cpt89\" (UID: \"bbab6794-a024-4b72-a8f2-2b9a4e85a3f2\") " pod="openshift-ingress/e2e-inf-a-39df6e-openshift-default-69b8d58b68-cpt89" Sep 10 13:29:22.494707 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:29:22.494687 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bbab6794-a024-4b72-a8f2-2b9a4e85a3f2-istio-podinfo\") pod \"e2e-inf-a-39df6e-openshift-default-69b8d58b68-cpt89\" (UID: \"bbab6794-a024-4b72-a8f2-2b9a4e85a3f2\") " pod="openshift-ingress/e2e-inf-a-39df6e-openshift-default-69b8d58b68-cpt89" Sep 10 13:29:22.501343 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:29:22.501312 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bbab6794-a024-4b72-a8f2-2b9a4e85a3f2-istio-token\") pod \"e2e-inf-a-39df6e-openshift-default-69b8d58b68-cpt89\" (UID: \"bbab6794-a024-4b72-a8f2-2b9a4e85a3f2\") " pod="openshift-ingress/e2e-inf-a-39df6e-openshift-default-69b8d58b68-cpt89" Sep 10 13:29:22.503913 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:29:22.503879 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zn9z4\" (UniqueName: \"kubernetes.io/projected/bbab6794-a024-4b72-a8f2-2b9a4e85a3f2-kube-api-access-zn9z4\") pod \"e2e-inf-a-39df6e-openshift-default-69b8d58b68-cpt89\" (UID: \"bbab6794-a024-4b72-a8f2-2b9a4e85a3f2\") " pod="openshift-ingress/e2e-inf-a-39df6e-openshift-default-69b8d58b68-cpt89" Sep 10 13:29:22.589056 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:29:22.589012 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-39df6e-openshift-default-69b8d58b68-cpt89" Sep 10 13:29:22.889715 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:29:22.889667 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-39df6e-openshift-default-69b8d58b68-cpt89"] Sep 10 13:29:22.894823 ip-10-0-139-174 kubenswrapper[2557]: W0910 13:29:22.894786 2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbbab6794_a024_4b72_a8f2_2b9a4e85a3f2.slice/crio-a514481c2e8452b14b76e4b5abe02f6bc6c4479d3bba35c20e92f120072df982 WatchSource:0}: Error finding container a514481c2e8452b14b76e4b5abe02f6bc6c4479d3bba35c20e92f120072df982: Status 404 returned error can't find the container with id a514481c2e8452b14b76e4b5abe02f6bc6c4479d3bba35c20e92f120072df982 Sep 10 13:29:22.896994 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:29:22.896922 2557 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892152Ki","pods":"250"} Sep 10 13:29:22.897079 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:29:22.897006 2557 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892152Ki","pods":"250"} Sep 10 13:29:22.897079 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:29:22.897062 2557 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892152Ki","pods":"250"} Sep 10 13:29:22.913196 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:29:22.913163 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-39df6e-openshift-default-69b8d58b68-cpt89" event={"ID":"bbab6794-a024-4b72-a8f2-2b9a4e85a3f2","Type":"ContainerStarted","Data":"a514481c2e8452b14b76e4b5abe02f6bc6c4479d3bba35c20e92f120072df982"} Sep 10 13:29:23.921521 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:29:23.921477 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-39df6e-openshift-default-69b8d58b68-cpt89" event={"ID":"bbab6794-a024-4b72-a8f2-2b9a4e85a3f2","Type":"ContainerStarted","Data":"1a7c02d08463ced64d15e6b76d7168971c3ae490800a4ff832023f1f3a88dfaf"} Sep 10 13:29:23.952811 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:29:23.952736 2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-39df6e-openshift-default-69b8d58b68-cpt89" podStartSLOduration=1.952714869 podStartE2EDuration="1.952714869s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 13:29:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 13:29:23.952350263 +0000 UTC m=+724.259344676" watchObservedRunningTime="2026-09-10 13:29:23.952714869 +0000 UTC m=+724.259709285" Sep 10 13:29:24.589756 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:29:24.589712 2557 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-39df6e-openshift-default-69b8d58b68-cpt89" Sep 10 13:29:24.595836 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:29:24.595802 2557 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-39df6e-openshift-default-69b8d58b68-cpt89" Sep 10 13:29:24.927671 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:29:24.927571 2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-39df6e-openshift-default-69b8d58b68-cpt89" Sep 10 13:29:24.928754 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:29:24.928736 2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-39df6e-openshift-default-69b8d58b68-cpt89" Sep 10 13:29:28.378023 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:29:28.377982 2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-39df6e-758db9c89f-d6wdl"] Sep 10 13:29:28.381949 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:29:28.381928 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-39df6e-758db9c89f-d6wdl" Sep 10 13:29:28.384363 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:29:28.384333 2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-a-39df6e\"" Sep 10 13:29:28.384363 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:29:28.384333 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-a-39df6e-dockercfg-ltpcq\"" Sep 10 13:29:28.394845 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:29:28.394814 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-39df6e-758db9c89f-d6wdl"] Sep 10 13:29:28.454301 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:29:28.454269 2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-39df6e-7bdcf56986-9czvg"] Sep 10 13:29:28.457897 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:29:28.457877 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-39df6e-7bdcf56986-9czvg" Sep 10 13:29:28.470012 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:29:28.469980 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-39df6e-7bdcf56986-9czvg"] Sep 10 13:29:28.559532 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:29:28.559486 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zf8xl\" (UniqueName: \"kubernetes.io/projected/516a2551-a59b-4435-8b6b-fdb75ca4d1d5-kube-api-access-zf8xl\") pod \"payload-processing-e2e-inf-a-39df6e-758db9c89f-d6wdl\" (UID: \"516a2551-a59b-4435-8b6b-fdb75ca4d1d5\") " pod="openshift-ingress/payload-processing-e2e-inf-a-39df6e-758db9c89f-d6wdl" Sep 10 13:29:28.559532 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:29:28.559530 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/516a2551-a59b-4435-8b6b-fdb75ca4d1d5-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-39df6e-758db9c89f-d6wdl\" (UID: \"516a2551-a59b-4435-8b6b-fdb75ca4d1d5\") " pod="openshift-ingress/payload-processing-e2e-inf-a-39df6e-758db9c89f-d6wdl" Sep 10 13:29:28.559855 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:29:28.559659 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5djkx\" (UniqueName: \"kubernetes.io/projected/ee2da889-3df3-4db7-abf0-9f5a75174e96-kube-api-access-5djkx\") pod \"payload-pre-processing-e2e-inf-a-39df6e-7bdcf56986-9czvg\" (UID: \"ee2da889-3df3-4db7-abf0-9f5a75174e96\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-39df6e-7bdcf56986-9czvg" Sep 10 13:29:28.559855 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:29:28.559701 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ee2da889-3df3-4db7-abf0-9f5a75174e96-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-39df6e-7bdcf56986-9czvg\" (UID: \"ee2da889-3df3-4db7-abf0-9f5a75174e96\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-39df6e-7bdcf56986-9czvg" Sep 10 13:29:28.661111 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:29:28.660996 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5djkx\" (UniqueName: \"kubernetes.io/projected/ee2da889-3df3-4db7-abf0-9f5a75174e96-kube-api-access-5djkx\") pod \"payload-pre-processing-e2e-inf-a-39df6e-7bdcf56986-9czvg\" (UID: \"ee2da889-3df3-4db7-abf0-9f5a75174e96\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-39df6e-7bdcf56986-9czvg" Sep 10 13:29:28.661111 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:29:28.661047 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ee2da889-3df3-4db7-abf0-9f5a75174e96-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-39df6e-7bdcf56986-9czvg\" (UID: \"ee2da889-3df3-4db7-abf0-9f5a75174e96\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-39df6e-7bdcf56986-9czvg" Sep 10 13:29:28.661111 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:29:28.661101 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zf8xl\" (UniqueName: \"kubernetes.io/projected/516a2551-a59b-4435-8b6b-fdb75ca4d1d5-kube-api-access-zf8xl\") pod \"payload-processing-e2e-inf-a-39df6e-758db9c89f-d6wdl\" (UID: \"516a2551-a59b-4435-8b6b-fdb75ca4d1d5\") " pod="openshift-ingress/payload-processing-e2e-inf-a-39df6e-758db9c89f-d6wdl" Sep 10 13:29:28.661464 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:29:28.661124 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/516a2551-a59b-4435-8b6b-fdb75ca4d1d5-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-39df6e-758db9c89f-d6wdl\" (UID: \"516a2551-a59b-4435-8b6b-fdb75ca4d1d5\") " pod="openshift-ingress/payload-processing-e2e-inf-a-39df6e-758db9c89f-d6wdl" Sep 10 13:29:28.661902 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:29:28.661877 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/516a2551-a59b-4435-8b6b-fdb75ca4d1d5-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-39df6e-758db9c89f-d6wdl\" (UID: \"516a2551-a59b-4435-8b6b-fdb75ca4d1d5\") " pod="openshift-ingress/payload-processing-e2e-inf-a-39df6e-758db9c89f-d6wdl" Sep 10 13:29:28.662062 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:29:28.662035 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ee2da889-3df3-4db7-abf0-9f5a75174e96-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-39df6e-7bdcf56986-9czvg\" (UID: \"ee2da889-3df3-4db7-abf0-9f5a75174e96\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-39df6e-7bdcf56986-9czvg" Sep 10 13:29:28.670243 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:29:28.670203 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zf8xl\" (UniqueName: \"kubernetes.io/projected/516a2551-a59b-4435-8b6b-fdb75ca4d1d5-kube-api-access-zf8xl\") pod \"payload-processing-e2e-inf-a-39df6e-758db9c89f-d6wdl\" (UID: \"516a2551-a59b-4435-8b6b-fdb75ca4d1d5\") " pod="openshift-ingress/payload-processing-e2e-inf-a-39df6e-758db9c89f-d6wdl" Sep 10 13:29:28.670474 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:29:28.670408 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5djkx\" (UniqueName: \"kubernetes.io/projected/ee2da889-3df3-4db7-abf0-9f5a75174e96-kube-api-access-5djkx\") pod \"payload-pre-processing-e2e-inf-a-39df6e-7bdcf56986-9czvg\" (UID: \"ee2da889-3df3-4db7-abf0-9f5a75174e96\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-39df6e-7bdcf56986-9czvg" Sep 10 13:29:28.695001 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:29:28.694960 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-39df6e-758db9c89f-d6wdl" Sep 10 13:29:28.769218 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:29:28.769004 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-39df6e-7bdcf56986-9czvg" Sep 10 13:29:28.981197 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:29:28.981163 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-39df6e-758db9c89f-d6wdl"] Sep 10 13:29:28.982319 ip-10-0-139-174 kubenswrapper[2557]: W0910 13:29:28.982298 2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod516a2551_a59b_4435_8b6b_fdb75ca4d1d5.slice/crio-ca15db838c2d527ccf094fbd403bf75dc8718e21fc46329d691db84f6362909a WatchSource:0}: Error finding container ca15db838c2d527ccf094fbd403bf75dc8718e21fc46329d691db84f6362909a: Status 404 returned error can't find the container with id ca15db838c2d527ccf094fbd403bf75dc8718e21fc46329d691db84f6362909a Sep 10 13:29:29.055224 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:29:29.055194 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-39df6e-7bdcf56986-9czvg"] Sep 10 13:29:29.056948 ip-10-0-139-174 kubenswrapper[2557]: W0910 13:29:29.056903 2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podee2da889_3df3_4db7_abf0_9f5a75174e96.slice/crio-240a7e80a1cebc31cdaaf7ef4f91a8d3479a48fffeacf7076fe6dc0793a4bd7f WatchSource:0}: Error finding container 240a7e80a1cebc31cdaaf7ef4f91a8d3479a48fffeacf7076fe6dc0793a4bd7f: Status 404 returned error can't find the container with id 240a7e80a1cebc31cdaaf7ef4f91a8d3479a48fffeacf7076fe6dc0793a4bd7f Sep 10 13:29:29.963576 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:29:29.963528 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-39df6e-7bdcf56986-9czvg" event={"ID":"ee2da889-3df3-4db7-abf0-9f5a75174e96","Type":"ContainerStarted","Data":"d531547914656f52029fd7af93bcbf2e6fb284576964283c0860da3bf39372d1"} Sep 10 13:29:29.964082 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:29:29.963646 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-39df6e-7bdcf56986-9czvg" event={"ID":"ee2da889-3df3-4db7-abf0-9f5a75174e96","Type":"ContainerStarted","Data":"240a7e80a1cebc31cdaaf7ef4f91a8d3479a48fffeacf7076fe6dc0793a4bd7f"} Sep 10 13:29:29.964082 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:29:29.963679 2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-39df6e-7bdcf56986-9czvg" Sep 10 13:29:29.966491 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:29:29.966434 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-39df6e-758db9c89f-d6wdl" event={"ID":"516a2551-a59b-4435-8b6b-fdb75ca4d1d5","Type":"ContainerStarted","Data":"7fa801e40c5a227a247bd78d5d448ba094bf0c1331f94dee6b65d6bf262d177e"} Sep 10 13:29:29.966691 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:29:29.966530 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-39df6e-758db9c89f-d6wdl" event={"ID":"516a2551-a59b-4435-8b6b-fdb75ca4d1d5","Type":"ContainerStarted","Data":"ca15db838c2d527ccf094fbd403bf75dc8718e21fc46329d691db84f6362909a"} Sep 10 13:29:29.966757 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:29:29.966720 2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-a-39df6e-758db9c89f-d6wdl" Sep 10 13:29:29.995358 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:29:29.995295 2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-39df6e-7bdcf56986-9czvg" podStartSLOduration=1.995276925 podStartE2EDuration="1.995276925s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 13:29:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 13:29:29.992939577 +0000 UTC m=+730.299933991" watchObservedRunningTime="2026-09-10 13:29:29.995276925 +0000 UTC m=+730.302271336" Sep 10 13:29:30.038654 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:29:30.038599 2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-a-39df6e-758db9c89f-d6wdl" podStartSLOduration=2.038578495 podStartE2EDuration="2.038578495s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 13:29:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 13:29:30.035370659 +0000 UTC m=+730.342365087" watchObservedRunningTime="2026-09-10 13:29:30.038578495 +0000 UTC m=+730.345572909" Sep 10 13:29:31.122637 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:29:31.122597 2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-autores-af59ca-openshift-default-887c6d76d-t4qb9"] Sep 10 13:29:31.126450 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:29:31.126412 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-autores-af59ca-openshift-default-887c6d76d-t4qb9" Sep 10 13:29:31.139353 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:29:31.139325 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-autores-af59ca-openshift-default-dockercfg-t9h2l\"" Sep 10 13:29:31.162305 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:29:31.162251 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-autores-af59ca-openshift-default-887c6d76d-t4qb9"] Sep 10 13:29:31.289332 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:29:31.289292 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/22c3a6a3-69e4-43db-8768-525668cb1928-istio-podinfo\") pod \"e2e-autores-af59ca-openshift-default-887c6d76d-t4qb9\" (UID: \"22c3a6a3-69e4-43db-8768-525668cb1928\") " pod="openshift-ingress/e2e-autores-af59ca-openshift-default-887c6d76d-t4qb9" Sep 10 13:29:31.289548 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:29:31.289347 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/22c3a6a3-69e4-43db-8768-525668cb1928-istiod-ca-cert\") pod \"e2e-autores-af59ca-openshift-default-887c6d76d-t4qb9\" (UID: \"22c3a6a3-69e4-43db-8768-525668cb1928\") " pod="openshift-ingress/e2e-autores-af59ca-openshift-default-887c6d76d-t4qb9" Sep 10 13:29:31.289548 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:29:31.289479 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/22c3a6a3-69e4-43db-8768-525668cb1928-istio-envoy\") pod \"e2e-autores-af59ca-openshift-default-887c6d76d-t4qb9\" (UID: \"22c3a6a3-69e4-43db-8768-525668cb1928\") " pod="openshift-ingress/e2e-autores-af59ca-openshift-default-887c6d76d-t4qb9" Sep 10 13:29:31.289627 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:29:31.289543 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wkd9n\" (UniqueName: \"kubernetes.io/projected/22c3a6a3-69e4-43db-8768-525668cb1928-kube-api-access-wkd9n\") pod \"e2e-autores-af59ca-openshift-default-887c6d76d-t4qb9\" (UID: \"22c3a6a3-69e4-43db-8768-525668cb1928\") " pod="openshift-ingress/e2e-autores-af59ca-openshift-default-887c6d76d-t4qb9" Sep 10 13:29:31.289627 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:29:31.289586 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/22c3a6a3-69e4-43db-8768-525668cb1928-credential-socket\") pod \"e2e-autores-af59ca-openshift-default-887c6d76d-t4qb9\" (UID: \"22c3a6a3-69e4-43db-8768-525668cb1928\") " pod="openshift-ingress/e2e-autores-af59ca-openshift-default-887c6d76d-t4qb9" Sep 10 13:29:31.289698 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:29:31.289645 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/22c3a6a3-69e4-43db-8768-525668cb1928-istio-data\") pod \"e2e-autores-af59ca-openshift-default-887c6d76d-t4qb9\" (UID: \"22c3a6a3-69e4-43db-8768-525668cb1928\") " pod="openshift-ingress/e2e-autores-af59ca-openshift-default-887c6d76d-t4qb9" Sep 10 13:29:31.289698 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:29:31.289676 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/22c3a6a3-69e4-43db-8768-525668cb1928-workload-certs\") pod \"e2e-autores-af59ca-openshift-default-887c6d76d-t4qb9\" (UID: \"22c3a6a3-69e4-43db-8768-525668cb1928\") " pod="openshift-ingress/e2e-autores-af59ca-openshift-default-887c6d76d-t4qb9" Sep 10 13:29:31.289758 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:29:31.289726 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/22c3a6a3-69e4-43db-8768-525668cb1928-workload-socket\") pod \"e2e-autores-af59ca-openshift-default-887c6d76d-t4qb9\" (UID: \"22c3a6a3-69e4-43db-8768-525668cb1928\") " pod="openshift-ingress/e2e-autores-af59ca-openshift-default-887c6d76d-t4qb9" Sep 10 13:29:31.289794 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:29:31.289766 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/22c3a6a3-69e4-43db-8768-525668cb1928-istio-token\") pod \"e2e-autores-af59ca-openshift-default-887c6d76d-t4qb9\" (UID: \"22c3a6a3-69e4-43db-8768-525668cb1928\") " pod="openshift-ingress/e2e-autores-af59ca-openshift-default-887c6d76d-t4qb9" Sep 10 13:29:31.391147 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:29:31.391034 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/22c3a6a3-69e4-43db-8768-525668cb1928-istiod-ca-cert\") pod \"e2e-autores-af59ca-openshift-default-887c6d76d-t4qb9\" (UID: \"22c3a6a3-69e4-43db-8768-525668cb1928\") " pod="openshift-ingress/e2e-autores-af59ca-openshift-default-887c6d76d-t4qb9" Sep 10 13:29:31.391147 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:29:31.391089 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/22c3a6a3-69e4-43db-8768-525668cb1928-istio-envoy\") pod \"e2e-autores-af59ca-openshift-default-887c6d76d-t4qb9\" (UID: \"22c3a6a3-69e4-43db-8768-525668cb1928\") " pod="openshift-ingress/e2e-autores-af59ca-openshift-default-887c6d76d-t4qb9" Sep 10 13:29:31.391147 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:29:31.391127 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wkd9n\" (UniqueName: \"kubernetes.io/projected/22c3a6a3-69e4-43db-8768-525668cb1928-kube-api-access-wkd9n\") pod \"e2e-autores-af59ca-openshift-default-887c6d76d-t4qb9\" (UID: \"22c3a6a3-69e4-43db-8768-525668cb1928\") " pod="openshift-ingress/e2e-autores-af59ca-openshift-default-887c6d76d-t4qb9" Sep 10 13:29:31.391147 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:29:31.391153 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/22c3a6a3-69e4-43db-8768-525668cb1928-credential-socket\") pod \"e2e-autores-af59ca-openshift-default-887c6d76d-t4qb9\" (UID: \"22c3a6a3-69e4-43db-8768-525668cb1928\") " pod="openshift-ingress/e2e-autores-af59ca-openshift-default-887c6d76d-t4qb9" Sep 10 13:29:31.391558 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:29:31.391192 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/22c3a6a3-69e4-43db-8768-525668cb1928-istio-data\") pod \"e2e-autores-af59ca-openshift-default-887c6d76d-t4qb9\" (UID: \"22c3a6a3-69e4-43db-8768-525668cb1928\") " pod="openshift-ingress/e2e-autores-af59ca-openshift-default-887c6d76d-t4qb9" Sep 10 13:29:31.391558 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:29:31.391222 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/22c3a6a3-69e4-43db-8768-525668cb1928-workload-certs\") pod \"e2e-autores-af59ca-openshift-default-887c6d76d-t4qb9\" (UID: \"22c3a6a3-69e4-43db-8768-525668cb1928\") " pod="openshift-ingress/e2e-autores-af59ca-openshift-default-887c6d76d-t4qb9" Sep 10 13:29:31.391558 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:29:31.391273 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/22c3a6a3-69e4-43db-8768-525668cb1928-workload-socket\") pod \"e2e-autores-af59ca-openshift-default-887c6d76d-t4qb9\" (UID: \"22c3a6a3-69e4-43db-8768-525668cb1928\") " pod="openshift-ingress/e2e-autores-af59ca-openshift-default-887c6d76d-t4qb9" Sep 10 13:29:31.391558 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:29:31.391309 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/22c3a6a3-69e4-43db-8768-525668cb1928-istio-token\") pod \"e2e-autores-af59ca-openshift-default-887c6d76d-t4qb9\" (UID: \"22c3a6a3-69e4-43db-8768-525668cb1928\") " pod="openshift-ingress/e2e-autores-af59ca-openshift-default-887c6d76d-t4qb9" Sep 10 13:29:31.391558 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:29:31.391367 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/22c3a6a3-69e4-43db-8768-525668cb1928-istio-podinfo\") pod \"e2e-autores-af59ca-openshift-default-887c6d76d-t4qb9\" (UID: \"22c3a6a3-69e4-43db-8768-525668cb1928\") " pod="openshift-ingress/e2e-autores-af59ca-openshift-default-887c6d76d-t4qb9" Sep 10 13:29:31.391877 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:29:31.391850 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/22c3a6a3-69e4-43db-8768-525668cb1928-workload-certs\") pod \"e2e-autores-af59ca-openshift-default-887c6d76d-t4qb9\" (UID: \"22c3a6a3-69e4-43db-8768-525668cb1928\") " pod="openshift-ingress/e2e-autores-af59ca-openshift-default-887c6d76d-t4qb9" Sep 10 13:29:31.392301 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:29:31.391930 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/22c3a6a3-69e4-43db-8768-525668cb1928-istiod-ca-cert\") pod \"e2e-autores-af59ca-openshift-default-887c6d76d-t4qb9\" (UID: \"22c3a6a3-69e4-43db-8768-525668cb1928\") " pod="openshift-ingress/e2e-autores-af59ca-openshift-default-887c6d76d-t4qb9" Sep 10 13:29:31.392301 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:29:31.392021 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/22c3a6a3-69e4-43db-8768-525668cb1928-credential-socket\") pod \"e2e-autores-af59ca-openshift-default-887c6d76d-t4qb9\" (UID: \"22c3a6a3-69e4-43db-8768-525668cb1928\") " pod="openshift-ingress/e2e-autores-af59ca-openshift-default-887c6d76d-t4qb9" Sep 10 13:29:31.392301 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:29:31.392154 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/22c3a6a3-69e4-43db-8768-525668cb1928-istio-data\") pod \"e2e-autores-af59ca-openshift-default-887c6d76d-t4qb9\" (UID: \"22c3a6a3-69e4-43db-8768-525668cb1928\") " pod="openshift-ingress/e2e-autores-af59ca-openshift-default-887c6d76d-t4qb9" Sep 10 13:29:31.392699 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:29:31.392521 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/22c3a6a3-69e4-43db-8768-525668cb1928-workload-socket\") pod \"e2e-autores-af59ca-openshift-default-887c6d76d-t4qb9\" (UID: \"22c3a6a3-69e4-43db-8768-525668cb1928\") " pod="openshift-ingress/e2e-autores-af59ca-openshift-default-887c6d76d-t4qb9" Sep 10 13:29:31.394278 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:29:31.394244 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/22c3a6a3-69e4-43db-8768-525668cb1928-istio-envoy\") pod \"e2e-autores-af59ca-openshift-default-887c6d76d-t4qb9\" (UID: \"22c3a6a3-69e4-43db-8768-525668cb1928\") " pod="openshift-ingress/e2e-autores-af59ca-openshift-default-887c6d76d-t4qb9" Sep 10 13:29:31.394384 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:29:31.394299 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/22c3a6a3-69e4-43db-8768-525668cb1928-istio-podinfo\") pod \"e2e-autores-af59ca-openshift-default-887c6d76d-t4qb9\" (UID: \"22c3a6a3-69e4-43db-8768-525668cb1928\") " pod="openshift-ingress/e2e-autores-af59ca-openshift-default-887c6d76d-t4qb9" Sep 10 13:29:31.408259 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:29:31.408188 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wkd9n\" (UniqueName: \"kubernetes.io/projected/22c3a6a3-69e4-43db-8768-525668cb1928-kube-api-access-wkd9n\") pod \"e2e-autores-af59ca-openshift-default-887c6d76d-t4qb9\" (UID: \"22c3a6a3-69e4-43db-8768-525668cb1928\") " pod="openshift-ingress/e2e-autores-af59ca-openshift-default-887c6d76d-t4qb9" Sep 10 13:29:31.409327 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:29:31.409294 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/22c3a6a3-69e4-43db-8768-525668cb1928-istio-token\") pod \"e2e-autores-af59ca-openshift-default-887c6d76d-t4qb9\" (UID: \"22c3a6a3-69e4-43db-8768-525668cb1928\") " pod="openshift-ingress/e2e-autores-af59ca-openshift-default-887c6d76d-t4qb9" Sep 10 13:29:31.438152 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:29:31.438109 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-autores-af59ca-openshift-default-887c6d76d-t4qb9" Sep 10 13:29:31.772937 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:29:31.772904 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-autores-af59ca-openshift-default-887c6d76d-t4qb9"] Sep 10 13:29:31.774091 ip-10-0-139-174 kubenswrapper[2557]: W0910 13:29:31.774055 2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod22c3a6a3_69e4_43db_8768_525668cb1928.slice/crio-016c34660714d2f5ac9f05e86ca451278e040d73451a86f6838c25f61deaf50a WatchSource:0}: Error finding container 016c34660714d2f5ac9f05e86ca451278e040d73451a86f6838c25f61deaf50a: Status 404 returned error can't find the container with id 016c34660714d2f5ac9f05e86ca451278e040d73451a86f6838c25f61deaf50a Sep 10 13:29:31.776778 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:29:31.776734 2557 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892152Ki","pods":"250"} Sep 10 13:29:31.776901 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:29:31.776816 2557 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892152Ki","pods":"250"} Sep 10 13:29:31.776901 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:29:31.776848 2557 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892152Ki","pods":"250"} Sep 10 13:29:31.986124 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:29:31.986080 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-autores-af59ca-openshift-default-887c6d76d-t4qb9" event={"ID":"22c3a6a3-69e4-43db-8768-525668cb1928","Type":"ContainerStarted","Data":"1135f5e5d87cb02f53d9744e567fffc5f706cae4f2503f2886bb9597b97f0349"} Sep 10 13:29:31.986346 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:29:31.986153 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-autores-af59ca-openshift-default-887c6d76d-t4qb9" event={"ID":"22c3a6a3-69e4-43db-8768-525668cb1928","Type":"ContainerStarted","Data":"016c34660714d2f5ac9f05e86ca451278e040d73451a86f6838c25f61deaf50a"} Sep 10 13:29:32.043718 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:29:32.043580 2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-autores-af59ca-openshift-default-887c6d76d-t4qb9" podStartSLOduration=1.043554071 podStartE2EDuration="1.043554071s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 13:29:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 13:29:32.040953226 +0000 UTC m=+732.347947653" watchObservedRunningTime="2026-09-10 13:29:32.043554071 +0000 UTC m=+732.350548487" Sep 10 13:29:32.439399 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:29:32.439296 2557 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-autores-af59ca-openshift-default-887c6d76d-t4qb9" Sep 10 13:29:32.447561 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:29:32.447527 2557 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-autores-af59ca-openshift-default-887c6d76d-t4qb9" Sep 10 13:29:32.991773 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:29:32.991727 2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-autores-af59ca-openshift-default-887c6d76d-t4qb9" Sep 10 13:29:32.992812 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:29:32.992791 2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-autores-af59ca-openshift-default-887c6d76d-t4qb9" Sep 10 13:29:34.315784 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:29:34.315737 2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ait-d2fb9c2a-openshift-default-689d78d8db-fxqrg"] Sep 10 13:29:34.320068 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:29:34.320039 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-d2fb9c2a-openshift-default-689d78d8db-fxqrg" Sep 10 13:29:34.336342 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:29:34.336313 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ait-d2fb9c2a-openshift-default-dockercfg-7s2vx\"" Sep 10 13:29:34.354452 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:29:34.354099 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-d2fb9c2a-openshift-default-689d78d8db-fxqrg"] Sep 10 13:29:34.423250 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:29:34.423207 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d605a297-fadf-4dc8-be1b-fc422c988207-istio-token\") pod \"e2e-ait-d2fb9c2a-openshift-default-689d78d8db-fxqrg\" (UID: \"d605a297-fadf-4dc8-be1b-fc422c988207\") " pod="openshift-ingress/e2e-ait-d2fb9c2a-openshift-default-689d78d8db-fxqrg" Sep 10 13:29:34.423406 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:29:34.423269 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d605a297-fadf-4dc8-be1b-fc422c988207-istio-envoy\") pod \"e2e-ait-d2fb9c2a-openshift-default-689d78d8db-fxqrg\" (UID: \"d605a297-fadf-4dc8-be1b-fc422c988207\") " pod="openshift-ingress/e2e-ait-d2fb9c2a-openshift-default-689d78d8db-fxqrg" Sep 10 13:29:34.423406 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:29:34.423324 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d605a297-fadf-4dc8-be1b-fc422c988207-istiod-ca-cert\") pod \"e2e-ait-d2fb9c2a-openshift-default-689d78d8db-fxqrg\" (UID: \"d605a297-fadf-4dc8-be1b-fc422c988207\") " pod="openshift-ingress/e2e-ait-d2fb9c2a-openshift-default-689d78d8db-fxqrg" Sep 10 13:29:34.423406 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:29:34.423396 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d605a297-fadf-4dc8-be1b-fc422c988207-workload-socket\") pod \"e2e-ait-d2fb9c2a-openshift-default-689d78d8db-fxqrg\" (UID: \"d605a297-fadf-4dc8-be1b-fc422c988207\") " pod="openshift-ingress/e2e-ait-d2fb9c2a-openshift-default-689d78d8db-fxqrg" Sep 10 13:29:34.423579 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:29:34.423455 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d605a297-fadf-4dc8-be1b-fc422c988207-credential-socket\") pod \"e2e-ait-d2fb9c2a-openshift-default-689d78d8db-fxqrg\" (UID: \"d605a297-fadf-4dc8-be1b-fc422c988207\") " pod="openshift-ingress/e2e-ait-d2fb9c2a-openshift-default-689d78d8db-fxqrg" Sep 10 13:29:34.423579 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:29:34.423551 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d605a297-fadf-4dc8-be1b-fc422c988207-istio-podinfo\") pod \"e2e-ait-d2fb9c2a-openshift-default-689d78d8db-fxqrg\" (UID: \"d605a297-fadf-4dc8-be1b-fc422c988207\") " pod="openshift-ingress/e2e-ait-d2fb9c2a-openshift-default-689d78d8db-fxqrg" Sep 10 13:29:34.423579 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:29:34.423571 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-plw22\" (UniqueName: \"kubernetes.io/projected/d605a297-fadf-4dc8-be1b-fc422c988207-kube-api-access-plw22\") pod \"e2e-ait-d2fb9c2a-openshift-default-689d78d8db-fxqrg\" (UID: \"d605a297-fadf-4dc8-be1b-fc422c988207\") " pod="openshift-ingress/e2e-ait-d2fb9c2a-openshift-default-689d78d8db-fxqrg" Sep 10 13:29:34.423689 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:29:34.423646 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d605a297-fadf-4dc8-be1b-fc422c988207-workload-certs\") pod \"e2e-ait-d2fb9c2a-openshift-default-689d78d8db-fxqrg\" (UID: \"d605a297-fadf-4dc8-be1b-fc422c988207\") " pod="openshift-ingress/e2e-ait-d2fb9c2a-openshift-default-689d78d8db-fxqrg" Sep 10 13:29:34.423689 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:29:34.423667 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d605a297-fadf-4dc8-be1b-fc422c988207-istio-data\") pod \"e2e-ait-d2fb9c2a-openshift-default-689d78d8db-fxqrg\" (UID: \"d605a297-fadf-4dc8-be1b-fc422c988207\") " pod="openshift-ingress/e2e-ait-d2fb9c2a-openshift-default-689d78d8db-fxqrg" Sep 10 13:29:34.525198 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:29:34.525144 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d605a297-fadf-4dc8-be1b-fc422c988207-istio-token\") pod \"e2e-ait-d2fb9c2a-openshift-default-689d78d8db-fxqrg\" (UID: \"d605a297-fadf-4dc8-be1b-fc422c988207\") " pod="openshift-ingress/e2e-ait-d2fb9c2a-openshift-default-689d78d8db-fxqrg" Sep 10 13:29:34.525370 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:29:34.525233 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d605a297-fadf-4dc8-be1b-fc422c988207-istio-envoy\") pod \"e2e-ait-d2fb9c2a-openshift-default-689d78d8db-fxqrg\" (UID: \"d605a297-fadf-4dc8-be1b-fc422c988207\") " pod="openshift-ingress/e2e-ait-d2fb9c2a-openshift-default-689d78d8db-fxqrg" Sep 10 13:29:34.525370 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:29:34.525260 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d605a297-fadf-4dc8-be1b-fc422c988207-istiod-ca-cert\") pod \"e2e-ait-d2fb9c2a-openshift-default-689d78d8db-fxqrg\" (UID: \"d605a297-fadf-4dc8-be1b-fc422c988207\") " pod="openshift-ingress/e2e-ait-d2fb9c2a-openshift-default-689d78d8db-fxqrg" Sep 10 13:29:34.525370 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:29:34.525283 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d605a297-fadf-4dc8-be1b-fc422c988207-workload-socket\") pod \"e2e-ait-d2fb9c2a-openshift-default-689d78d8db-fxqrg\" (UID: \"d605a297-fadf-4dc8-be1b-fc422c988207\") " pod="openshift-ingress/e2e-ait-d2fb9c2a-openshift-default-689d78d8db-fxqrg" Sep 10 13:29:34.525370 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:29:34.525306 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d605a297-fadf-4dc8-be1b-fc422c988207-credential-socket\") pod \"e2e-ait-d2fb9c2a-openshift-default-689d78d8db-fxqrg\" (UID: \"d605a297-fadf-4dc8-be1b-fc422c988207\") " pod="openshift-ingress/e2e-ait-d2fb9c2a-openshift-default-689d78d8db-fxqrg" Sep 10 13:29:34.525370 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:29:34.525344 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d605a297-fadf-4dc8-be1b-fc422c988207-istio-podinfo\") pod \"e2e-ait-d2fb9c2a-openshift-default-689d78d8db-fxqrg\" (UID: \"d605a297-fadf-4dc8-be1b-fc422c988207\") " pod="openshift-ingress/e2e-ait-d2fb9c2a-openshift-default-689d78d8db-fxqrg" Sep 10 13:29:34.525370 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:29:34.525361 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-plw22\" (UniqueName: \"kubernetes.io/projected/d605a297-fadf-4dc8-be1b-fc422c988207-kube-api-access-plw22\") pod \"e2e-ait-d2fb9c2a-openshift-default-689d78d8db-fxqrg\" (UID: \"d605a297-fadf-4dc8-be1b-fc422c988207\") " pod="openshift-ingress/e2e-ait-d2fb9c2a-openshift-default-689d78d8db-fxqrg" Sep 10 13:29:34.525617 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:29:34.525435 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d605a297-fadf-4dc8-be1b-fc422c988207-workload-certs\") pod \"e2e-ait-d2fb9c2a-openshift-default-689d78d8db-fxqrg\" (UID: \"d605a297-fadf-4dc8-be1b-fc422c988207\") " pod="openshift-ingress/e2e-ait-d2fb9c2a-openshift-default-689d78d8db-fxqrg" Sep 10 13:29:34.525617 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:29:34.525451 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d605a297-fadf-4dc8-be1b-fc422c988207-istio-data\") pod \"e2e-ait-d2fb9c2a-openshift-default-689d78d8db-fxqrg\" (UID: \"d605a297-fadf-4dc8-be1b-fc422c988207\") " pod="openshift-ingress/e2e-ait-d2fb9c2a-openshift-default-689d78d8db-fxqrg" Sep 10 13:29:34.525919 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:29:34.525893 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d605a297-fadf-4dc8-be1b-fc422c988207-credential-socket\") pod \"e2e-ait-d2fb9c2a-openshift-default-689d78d8db-fxqrg\" (UID: \"d605a297-fadf-4dc8-be1b-fc422c988207\") " pod="openshift-ingress/e2e-ait-d2fb9c2a-openshift-default-689d78d8db-fxqrg" Sep 10 13:29:34.526092 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:29:34.526073 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d605a297-fadf-4dc8-be1b-fc422c988207-workload-socket\") pod \"e2e-ait-d2fb9c2a-openshift-default-689d78d8db-fxqrg\" (UID: \"d605a297-fadf-4dc8-be1b-fc422c988207\") " pod="openshift-ingress/e2e-ait-d2fb9c2a-openshift-default-689d78d8db-fxqrg" Sep 10 13:29:34.526181 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:29:34.526155 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d605a297-fadf-4dc8-be1b-fc422c988207-workload-certs\") pod \"e2e-ait-d2fb9c2a-openshift-default-689d78d8db-fxqrg\" (UID: \"d605a297-fadf-4dc8-be1b-fc422c988207\") " pod="openshift-ingress/e2e-ait-d2fb9c2a-openshift-default-689d78d8db-fxqrg" Sep 10 13:29:34.526463 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:29:34.526443 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d605a297-fadf-4dc8-be1b-fc422c988207-istio-data\") pod \"e2e-ait-d2fb9c2a-openshift-default-689d78d8db-fxqrg\" (UID: \"d605a297-fadf-4dc8-be1b-fc422c988207\") " pod="openshift-ingress/e2e-ait-d2fb9c2a-openshift-default-689d78d8db-fxqrg" Sep 10 13:29:34.526719 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:29:34.526700 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d605a297-fadf-4dc8-be1b-fc422c988207-istiod-ca-cert\") pod \"e2e-ait-d2fb9c2a-openshift-default-689d78d8db-fxqrg\" (UID: \"d605a297-fadf-4dc8-be1b-fc422c988207\") " pod="openshift-ingress/e2e-ait-d2fb9c2a-openshift-default-689d78d8db-fxqrg" Sep 10 13:29:34.528385 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:29:34.528279 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d605a297-fadf-4dc8-be1b-fc422c988207-istio-podinfo\") pod \"e2e-ait-d2fb9c2a-openshift-default-689d78d8db-fxqrg\" (UID: \"d605a297-fadf-4dc8-be1b-fc422c988207\") " pod="openshift-ingress/e2e-ait-d2fb9c2a-openshift-default-689d78d8db-fxqrg" Sep 10 13:29:34.528385 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:29:34.528335 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d605a297-fadf-4dc8-be1b-fc422c988207-istio-envoy\") pod \"e2e-ait-d2fb9c2a-openshift-default-689d78d8db-fxqrg\" (UID: \"d605a297-fadf-4dc8-be1b-fc422c988207\") " pod="openshift-ingress/e2e-ait-d2fb9c2a-openshift-default-689d78d8db-fxqrg" Sep 10 13:29:34.561491 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:29:34.561457 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d605a297-fadf-4dc8-be1b-fc422c988207-istio-token\") pod \"e2e-ait-d2fb9c2a-openshift-default-689d78d8db-fxqrg\" (UID: \"d605a297-fadf-4dc8-be1b-fc422c988207\") " pod="openshift-ingress/e2e-ait-d2fb9c2a-openshift-default-689d78d8db-fxqrg" Sep 10 13:29:34.561711 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:29:34.561683 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-plw22\" (UniqueName: \"kubernetes.io/projected/d605a297-fadf-4dc8-be1b-fc422c988207-kube-api-access-plw22\") pod \"e2e-ait-d2fb9c2a-openshift-default-689d78d8db-fxqrg\" (UID: \"d605a297-fadf-4dc8-be1b-fc422c988207\") " pod="openshift-ingress/e2e-ait-d2fb9c2a-openshift-default-689d78d8db-fxqrg" Sep 10 13:29:34.634813 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:29:34.634712 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-d2fb9c2a-openshift-default-689d78d8db-fxqrg" Sep 10 13:29:34.954788 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:29:34.954742 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-d2fb9c2a-openshift-default-689d78d8db-fxqrg"] Sep 10 13:29:34.961156 ip-10-0-139-174 kubenswrapper[2557]: W0910 13:29:34.961096 2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd605a297_fadf_4dc8_be1b_fc422c988207.slice/crio-1a2d778c40f97870d4cd5ab42c30395ed214c937264b5996f8dc6a9d5f10a8e5 WatchSource:0}: Error finding container 1a2d778c40f97870d4cd5ab42c30395ed214c937264b5996f8dc6a9d5f10a8e5: Status 404 returned error can't find the container with id 1a2d778c40f97870d4cd5ab42c30395ed214c937264b5996f8dc6a9d5f10a8e5 Sep 10 13:29:34.964014 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:29:34.963953 2557 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892152Ki","pods":"250"} Sep 10 13:29:34.964132 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:29:34.964073 2557 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892152Ki","pods":"250"} Sep 10 13:29:34.964132 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:29:34.964121 2557 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892152Ki","pods":"250"} Sep 10 13:29:35.005123 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:29:35.005070 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-d2fb9c2a-openshift-default-689d78d8db-fxqrg" event={"ID":"d605a297-fadf-4dc8-be1b-fc422c988207","Type":"ContainerStarted","Data":"1a2d778c40f97870d4cd5ab42c30395ed214c937264b5996f8dc6a9d5f10a8e5"} Sep 10 13:29:36.016525 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:29:36.016251 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-d2fb9c2a-openshift-default-689d78d8db-fxqrg" event={"ID":"d605a297-fadf-4dc8-be1b-fc422c988207","Type":"ContainerStarted","Data":"65fc3cadea708bae2c5dae204ea39db6e92f9d38eca8882749b4ca2ae93b83a3"} Sep 10 13:29:36.635752 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:29:36.635712 2557 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-d2fb9c2a-openshift-default-689d78d8db-fxqrg" Sep 10 13:29:36.642306 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:29:36.642272 2557 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-d2fb9c2a-openshift-default-689d78d8db-fxqrg" Sep 10 13:29:36.682318 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:29:36.682240 2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-ait-d2fb9c2a-openshift-default-689d78d8db-fxqrg" podStartSLOduration=2.682215278 podStartE2EDuration="2.682215278s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 13:29:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 13:29:36.064581448 +0000 UTC m=+736.371575863" watchObservedRunningTime="2026-09-10 13:29:36.682215278 +0000 UTC m=+736.989209684" Sep 10 13:29:37.022481 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:29:37.022451 2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-d2fb9c2a-openshift-default-689d78d8db-fxqrg" Sep 10 13:29:37.023720 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:29:37.023697 2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-d2fb9c2a-openshift-default-689d78d8db-fxqrg" Sep 10 13:29:39.559158 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:29:39.559111 2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-d2fb9c2a-846bdcb669-kh87w"] Sep 10 13:29:39.563576 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:29:39.563549 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-d2fb9c2a-846bdcb669-kh87w" Sep 10 13:29:39.565994 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:29:39.565966 2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ait-d2fb9c2a\"" Sep 10 13:29:39.582764 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:29:39.582674 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jhmt8\" (UniqueName: \"kubernetes.io/projected/1883269d-5793-47cf-b93a-35658a57e489-kube-api-access-jhmt8\") pod \"payload-processing-e2e-ait-d2fb9c2a-846bdcb669-kh87w\" (UID: \"1883269d-5793-47cf-b93a-35658a57e489\") " pod="openshift-ingress/payload-processing-e2e-ait-d2fb9c2a-846bdcb669-kh87w" Sep 10 13:29:39.582976 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:29:39.582886 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1883269d-5793-47cf-b93a-35658a57e489-plugins-config-volume\") pod \"payload-processing-e2e-ait-d2fb9c2a-846bdcb669-kh87w\" (UID: \"1883269d-5793-47cf-b93a-35658a57e489\") " pod="openshift-ingress/payload-processing-e2e-ait-d2fb9c2a-846bdcb669-kh87w" Sep 10 13:29:39.599332 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:29:39.599297 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-d2fb9c2a-846bdcb669-kh87w"] Sep 10 13:29:39.684452 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:29:39.684371 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jhmt8\" (UniqueName: \"kubernetes.io/projected/1883269d-5793-47cf-b93a-35658a57e489-kube-api-access-jhmt8\") pod \"payload-processing-e2e-ait-d2fb9c2a-846bdcb669-kh87w\" (UID: \"1883269d-5793-47cf-b93a-35658a57e489\") " pod="openshift-ingress/payload-processing-e2e-ait-d2fb9c2a-846bdcb669-kh87w" Sep 10 13:29:39.684674 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:29:39.684569 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1883269d-5793-47cf-b93a-35658a57e489-plugins-config-volume\") pod \"payload-processing-e2e-ait-d2fb9c2a-846bdcb669-kh87w\" (UID: \"1883269d-5793-47cf-b93a-35658a57e489\") " pod="openshift-ingress/payload-processing-e2e-ait-d2fb9c2a-846bdcb669-kh87w" Sep 10 13:29:39.685798 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:29:39.685762 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1883269d-5793-47cf-b93a-35658a57e489-plugins-config-volume\") pod \"payload-processing-e2e-ait-d2fb9c2a-846bdcb669-kh87w\" (UID: \"1883269d-5793-47cf-b93a-35658a57e489\") " pod="openshift-ingress/payload-processing-e2e-ait-d2fb9c2a-846bdcb669-kh87w" Sep 10 13:29:39.704585 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:29:39.704547 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jhmt8\" (UniqueName: \"kubernetes.io/projected/1883269d-5793-47cf-b93a-35658a57e489-kube-api-access-jhmt8\") pod \"payload-processing-e2e-ait-d2fb9c2a-846bdcb669-kh87w\" (UID: \"1883269d-5793-47cf-b93a-35658a57e489\") " pod="openshift-ingress/payload-processing-e2e-ait-d2fb9c2a-846bdcb669-kh87w" Sep 10 13:29:39.875675 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:29:39.875586 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-d2fb9c2a-846bdcb669-kh87w" Sep 10 13:29:40.168835 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:29:40.168555 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-d2fb9c2a-846bdcb669-kh87w"] Sep 10 13:29:40.170759 ip-10-0-139-174 kubenswrapper[2557]: W0910 13:29:40.170720 2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1883269d_5793_47cf_b93a_35658a57e489.slice/crio-9b6e5b39772ab78f5ec84e79b245ad6fd7c0b68929847a3a578505d68c75c7e3 WatchSource:0}: Error finding container 9b6e5b39772ab78f5ec84e79b245ad6fd7c0b68929847a3a578505d68c75c7e3: Status 404 returned error can't find the container with id 9b6e5b39772ab78f5ec84e79b245ad6fd7c0b68929847a3a578505d68c75c7e3 Sep 10 13:29:40.197842 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:29:40.197802 2557 patch_prober.go:28] interesting pod/e2e-inf-a-39df6e-openshift-default-69b8d58b68-cpt89 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.42:15021/healthz/ready\": read tcp 10.134.0.2:60928->10.134.0.42:15021: read: connection reset by peer" start-of-body= Sep 10 13:29:40.198327 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:29:40.197881 2557 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-a-39df6e-openshift-default-69b8d58b68-cpt89" podUID="bbab6794-a024-4b72-a8f2-2b9a4e85a3f2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.42:15021/healthz/ready\": read tcp 10.134.0.2:60928->10.134.0.42:15021: read: connection reset by peer" Sep 10 13:29:40.978158 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:29:40.978071 2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-39df6e-7bdcf56986-9czvg" Sep 10 13:29:40.978631 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:29:40.978160 2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-a-39df6e-758db9c89f-d6wdl" Sep 10 13:29:41.049025 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:29:41.048993 2557 generic.go:356] "Generic (PLEG): container finished" podID="bbab6794-a024-4b72-a8f2-2b9a4e85a3f2" containerID="1a7c02d08463ced64d15e6b76d7168971c3ae490800a4ff832023f1f3a88dfaf" exitCode=0 Sep 10 13:29:41.049231 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:29:41.049069 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-39df6e-openshift-default-69b8d58b68-cpt89" event={"ID":"bbab6794-a024-4b72-a8f2-2b9a4e85a3f2","Type":"ContainerDied","Data":"1a7c02d08463ced64d15e6b76d7168971c3ae490800a4ff832023f1f3a88dfaf"} Sep 10 13:29:41.049731 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:29:41.049708 2557 scope.go:117] "RemoveContainer" containerID="1a7c02d08463ced64d15e6b76d7168971c3ae490800a4ff832023f1f3a88dfaf" Sep 10 13:29:41.051037 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:29:41.051003 2557 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892152Ki","pods":"250"} Sep 10 13:29:41.051172 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:29:41.051091 2557 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892152Ki","pods":"250"} Sep 10 13:29:41.051172 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:29:41.051144 2557 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892152Ki","pods":"250"} Sep 10 13:29:41.051172 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:29:41.051151 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-d2fb9c2a-846bdcb669-kh87w" event={"ID":"1883269d-5793-47cf-b93a-35658a57e489","Type":"ContainerStarted","Data":"fa3967ca67ae7375d11b8257ed78e9df9e561c87fe613032b4185bc9d0c4d506"} Sep 10 13:29:41.051308 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:29:41.051210 2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ait-d2fb9c2a-846bdcb669-kh87w" Sep 10 13:29:41.051308 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:29:41.051230 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-d2fb9c2a-846bdcb669-kh87w" event={"ID":"1883269d-5793-47cf-b93a-35658a57e489","Type":"ContainerStarted","Data":"9b6e5b39772ab78f5ec84e79b245ad6fd7c0b68929847a3a578505d68c75c7e3"} Sep 10 13:29:41.091110 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:29:41.091047 2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ait-d2fb9c2a-846bdcb669-kh87w" podStartSLOduration=2.091021471 podStartE2EDuration="2.091021471s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 13:29:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 13:29:41.088965552 +0000 UTC m=+741.395959961" watchObservedRunningTime="2026-09-10 13:29:41.091021471 +0000 UTC m=+741.398015887" Sep 10 13:29:42.068217 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:29:42.068173 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-39df6e-openshift-default-69b8d58b68-cpt89" event={"ID":"bbab6794-a024-4b72-a8f2-2b9a4e85a3f2","Type":"ContainerStarted","Data":"750b8fc01729c33da6eb4abe1fdfed8db3dca6ba4bcb4e103f01be459caeb32a"} Sep 10 13:29:42.589753 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:29:42.589713 2557 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-39df6e-openshift-default-69b8d58b68-cpt89" Sep 10 13:29:42.589938 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:29:42.589898 2557 patch_prober.go:28] interesting pod/e2e-inf-a-39df6e-openshift-default-69b8d58b68-cpt89 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.42:15021/healthz/ready\": dial tcp 10.134.0.42:15021: connect: connection refused" start-of-body= Sep 10 13:29:42.589986 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:29:42.589949 2557 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-39df6e-openshift-default-69b8d58b68-cpt89" podUID="bbab6794-a024-4b72-a8f2-2b9a4e85a3f2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.42:15021/healthz/ready\": dial tcp 10.134.0.42:15021: connect: connection refused" Sep 10 13:29:43.590278 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:29:43.590238 2557 patch_prober.go:28] interesting pod/e2e-inf-a-39df6e-openshift-default-69b8d58b68-cpt89 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.42:15021/healthz/ready\": dial tcp 10.134.0.42:15021: connect: connection refused" start-of-body= Sep 10 13:29:43.590806 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:29:43.590300 2557 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-39df6e-openshift-default-69b8d58b68-cpt89" podUID="bbab6794-a024-4b72-a8f2-2b9a4e85a3f2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.42:15021/healthz/ready\": dial tcp 10.134.0.42:15021: connect: connection refused" Sep 10 13:29:44.590306 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:29:44.590271 2557 patch_prober.go:28] interesting pod/e2e-inf-a-39df6e-openshift-default-69b8d58b68-cpt89 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.42:15021/healthz/ready\": dial tcp 10.134.0.42:15021: connect: connection refused" start-of-body= Sep 10 13:29:44.590719 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:29:44.590330 2557 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-39df6e-openshift-default-69b8d58b68-cpt89" podUID="bbab6794-a024-4b72-a8f2-2b9a4e85a3f2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.42:15021/healthz/ready\": dial tcp 10.134.0.42:15021: connect: connection refused" Sep 10 13:29:45.591186 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:29:45.591142 2557 patch_prober.go:28] interesting pod/e2e-inf-a-39df6e-openshift-default-69b8d58b68-cpt89 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.42:15021/healthz/ready\": dial tcp 10.134.0.42:15021: connect: connection refused" start-of-body= Sep 10 13:29:45.591858 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:29:45.591820 2557 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-39df6e-openshift-default-69b8d58b68-cpt89" podUID="bbab6794-a024-4b72-a8f2-2b9a4e85a3f2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.42:15021/healthz/ready\": dial tcp 10.134.0.42:15021: connect: connection refused" Sep 10 13:29:46.590098 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:29:46.590060 2557 patch_prober.go:28] interesting pod/e2e-inf-a-39df6e-openshift-default-69b8d58b68-cpt89 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.42:15021/healthz/ready\": dial tcp 10.134.0.42:15021: connect: connection refused" start-of-body= Sep 10 13:29:46.590446 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:29:46.590388 2557 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-39df6e-openshift-default-69b8d58b68-cpt89" podUID="bbab6794-a024-4b72-a8f2-2b9a4e85a3f2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.42:15021/healthz/ready\": dial tcp 10.134.0.42:15021: connect: connection refused" Sep 10 13:29:47.597005 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:29:47.596960 2557 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-39df6e-openshift-default-69b8d58b68-cpt89" Sep 10 13:29:47.597842 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:29:47.597818 2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-39df6e-openshift-default-69b8d58b68-cpt89" Sep 10 13:29:47.598652 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:29:47.598634 2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-39df6e-openshift-default-69b8d58b68-cpt89" Sep 10 13:29:51.802513 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:29:51.802474 2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-b-6beee4-openshift-default-66d46f8786-2vth6"] Sep 10 13:29:51.805922 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:29:51.805895 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-6beee4-openshift-default-66d46f8786-2vth6" Sep 10 13:29:51.808339 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:29:51.808224 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-b-6beee4-openshift-default-dockercfg-fmwr8\"" Sep 10 13:29:51.821597 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:29:51.821563 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-6beee4-openshift-default-66d46f8786-2vth6"] Sep 10 13:29:51.938213 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:29:51.938174 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b23f2560-5aa1-43b0-b0d9-674c6b0eb707-workload-certs\") pod \"e2e-inf-b-6beee4-openshift-default-66d46f8786-2vth6\" (UID: \"b23f2560-5aa1-43b0-b0d9-674c6b0eb707\") " pod="openshift-ingress/e2e-inf-b-6beee4-openshift-default-66d46f8786-2vth6" Sep 10 13:29:51.938458 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:29:51.938408 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b23f2560-5aa1-43b0-b0d9-674c6b0eb707-istio-podinfo\") pod \"e2e-inf-b-6beee4-openshift-default-66d46f8786-2vth6\" (UID: \"b23f2560-5aa1-43b0-b0d9-674c6b0eb707\") " pod="openshift-ingress/e2e-inf-b-6beee4-openshift-default-66d46f8786-2vth6" Sep 10 13:29:51.938681 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:29:51.938621 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b23f2560-5aa1-43b0-b0d9-674c6b0eb707-istio-data\") pod \"e2e-inf-b-6beee4-openshift-default-66d46f8786-2vth6\" (UID: \"b23f2560-5aa1-43b0-b0d9-674c6b0eb707\") " pod="openshift-ingress/e2e-inf-b-6beee4-openshift-default-66d46f8786-2vth6" Sep 10 13:29:51.938939 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:29:51.938887 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b23f2560-5aa1-43b0-b0d9-674c6b0eb707-istio-envoy\") pod \"e2e-inf-b-6beee4-openshift-default-66d46f8786-2vth6\" (UID: \"b23f2560-5aa1-43b0-b0d9-674c6b0eb707\") " pod="openshift-ingress/e2e-inf-b-6beee4-openshift-default-66d46f8786-2vth6" Sep 10 13:29:51.939125 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:29:51.939058 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-htr6w\" (UniqueName: \"kubernetes.io/projected/b23f2560-5aa1-43b0-b0d9-674c6b0eb707-kube-api-access-htr6w\") pod \"e2e-inf-b-6beee4-openshift-default-66d46f8786-2vth6\" (UID: \"b23f2560-5aa1-43b0-b0d9-674c6b0eb707\") " pod="openshift-ingress/e2e-inf-b-6beee4-openshift-default-66d46f8786-2vth6" Sep 10 13:29:51.939276 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:29:51.939235 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b23f2560-5aa1-43b0-b0d9-674c6b0eb707-istiod-ca-cert\") pod \"e2e-inf-b-6beee4-openshift-default-66d46f8786-2vth6\" (UID: \"b23f2560-5aa1-43b0-b0d9-674c6b0eb707\") " pod="openshift-ingress/e2e-inf-b-6beee4-openshift-default-66d46f8786-2vth6" Sep 10 13:29:51.939545 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:29:51.939479 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b23f2560-5aa1-43b0-b0d9-674c6b0eb707-credential-socket\") pod \"e2e-inf-b-6beee4-openshift-default-66d46f8786-2vth6\" (UID: \"b23f2560-5aa1-43b0-b0d9-674c6b0eb707\") " pod="openshift-ingress/e2e-inf-b-6beee4-openshift-default-66d46f8786-2vth6" Sep 10 13:29:51.940935 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:29:51.940747 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b23f2560-5aa1-43b0-b0d9-674c6b0eb707-workload-socket\") pod \"e2e-inf-b-6beee4-openshift-default-66d46f8786-2vth6\" (UID: \"b23f2560-5aa1-43b0-b0d9-674c6b0eb707\") " pod="openshift-ingress/e2e-inf-b-6beee4-openshift-default-66d46f8786-2vth6" Sep 10 13:29:51.940935 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:29:51.940824 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b23f2560-5aa1-43b0-b0d9-674c6b0eb707-istio-token\") pod \"e2e-inf-b-6beee4-openshift-default-66d46f8786-2vth6\" (UID: \"b23f2560-5aa1-43b0-b0d9-674c6b0eb707\") " pod="openshift-ingress/e2e-inf-b-6beee4-openshift-default-66d46f8786-2vth6" Sep 10 13:29:52.041685 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:29:52.041644 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b23f2560-5aa1-43b0-b0d9-674c6b0eb707-istio-token\") pod \"e2e-inf-b-6beee4-openshift-default-66d46f8786-2vth6\" (UID: \"b23f2560-5aa1-43b0-b0d9-674c6b0eb707\") " pod="openshift-ingress/e2e-inf-b-6beee4-openshift-default-66d46f8786-2vth6" Sep 10 13:29:52.041896 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:29:52.041746 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b23f2560-5aa1-43b0-b0d9-674c6b0eb707-workload-certs\") pod \"e2e-inf-b-6beee4-openshift-default-66d46f8786-2vth6\" (UID: \"b23f2560-5aa1-43b0-b0d9-674c6b0eb707\") " pod="openshift-ingress/e2e-inf-b-6beee4-openshift-default-66d46f8786-2vth6" Sep 10 13:29:52.041896 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:29:52.041790 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b23f2560-5aa1-43b0-b0d9-674c6b0eb707-istio-podinfo\") pod \"e2e-inf-b-6beee4-openshift-default-66d46f8786-2vth6\" (UID: \"b23f2560-5aa1-43b0-b0d9-674c6b0eb707\") " pod="openshift-ingress/e2e-inf-b-6beee4-openshift-default-66d46f8786-2vth6" Sep 10 13:29:52.041896 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:29:52.041846 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b23f2560-5aa1-43b0-b0d9-674c6b0eb707-istio-data\") pod \"e2e-inf-b-6beee4-openshift-default-66d46f8786-2vth6\" (UID: \"b23f2560-5aa1-43b0-b0d9-674c6b0eb707\") " pod="openshift-ingress/e2e-inf-b-6beee4-openshift-default-66d46f8786-2vth6" Sep 10 13:29:52.042051 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:29:52.041905 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b23f2560-5aa1-43b0-b0d9-674c6b0eb707-istio-envoy\") pod \"e2e-inf-b-6beee4-openshift-default-66d46f8786-2vth6\" (UID: \"b23f2560-5aa1-43b0-b0d9-674c6b0eb707\") " pod="openshift-ingress/e2e-inf-b-6beee4-openshift-default-66d46f8786-2vth6" Sep 10 13:29:52.042051 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:29:52.041950 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-htr6w\" (UniqueName: \"kubernetes.io/projected/b23f2560-5aa1-43b0-b0d9-674c6b0eb707-kube-api-access-htr6w\") pod \"e2e-inf-b-6beee4-openshift-default-66d46f8786-2vth6\" (UID: \"b23f2560-5aa1-43b0-b0d9-674c6b0eb707\") " pod="openshift-ingress/e2e-inf-b-6beee4-openshift-default-66d46f8786-2vth6" Sep 10 13:29:52.042051 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:29:52.041983 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b23f2560-5aa1-43b0-b0d9-674c6b0eb707-istiod-ca-cert\") pod \"e2e-inf-b-6beee4-openshift-default-66d46f8786-2vth6\" (UID: \"b23f2560-5aa1-43b0-b0d9-674c6b0eb707\") " pod="openshift-ingress/e2e-inf-b-6beee4-openshift-default-66d46f8786-2vth6" Sep 10 13:29:52.042051 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:29:52.042020 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b23f2560-5aa1-43b0-b0d9-674c6b0eb707-credential-socket\") pod \"e2e-inf-b-6beee4-openshift-default-66d46f8786-2vth6\" (UID: \"b23f2560-5aa1-43b0-b0d9-674c6b0eb707\") " pod="openshift-ingress/e2e-inf-b-6beee4-openshift-default-66d46f8786-2vth6" Sep 10 13:29:52.042226 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:29:52.042089 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b23f2560-5aa1-43b0-b0d9-674c6b0eb707-workload-socket\") pod \"e2e-inf-b-6beee4-openshift-default-66d46f8786-2vth6\" (UID: \"b23f2560-5aa1-43b0-b0d9-674c6b0eb707\") " pod="openshift-ingress/e2e-inf-b-6beee4-openshift-default-66d46f8786-2vth6" Sep 10 13:29:52.042226 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:29:52.042214 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b23f2560-5aa1-43b0-b0d9-674c6b0eb707-workload-certs\") pod \"e2e-inf-b-6beee4-openshift-default-66d46f8786-2vth6\" (UID: \"b23f2560-5aa1-43b0-b0d9-674c6b0eb707\") " pod="openshift-ingress/e2e-inf-b-6beee4-openshift-default-66d46f8786-2vth6" Sep 10 13:29:52.042324 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:29:52.042291 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b23f2560-5aa1-43b0-b0d9-674c6b0eb707-istio-data\") pod \"e2e-inf-b-6beee4-openshift-default-66d46f8786-2vth6\" (UID: \"b23f2560-5aa1-43b0-b0d9-674c6b0eb707\") " pod="openshift-ingress/e2e-inf-b-6beee4-openshift-default-66d46f8786-2vth6" Sep 10 13:29:52.042635 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:29:52.042606 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b23f2560-5aa1-43b0-b0d9-674c6b0eb707-credential-socket\") pod \"e2e-inf-b-6beee4-openshift-default-66d46f8786-2vth6\" (UID: \"b23f2560-5aa1-43b0-b0d9-674c6b0eb707\") " pod="openshift-ingress/e2e-inf-b-6beee4-openshift-default-66d46f8786-2vth6" Sep 10 13:29:52.042738 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:29:52.042686 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b23f2560-5aa1-43b0-b0d9-674c6b0eb707-workload-socket\") pod \"e2e-inf-b-6beee4-openshift-default-66d46f8786-2vth6\" (UID: \"b23f2560-5aa1-43b0-b0d9-674c6b0eb707\") " pod="openshift-ingress/e2e-inf-b-6beee4-openshift-default-66d46f8786-2vth6" Sep 10 13:29:52.042782 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:29:52.042759 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b23f2560-5aa1-43b0-b0d9-674c6b0eb707-istiod-ca-cert\") pod \"e2e-inf-b-6beee4-openshift-default-66d46f8786-2vth6\" (UID: \"b23f2560-5aa1-43b0-b0d9-674c6b0eb707\") " pod="openshift-ingress/e2e-inf-b-6beee4-openshift-default-66d46f8786-2vth6" Sep 10 13:29:52.045050 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:29:52.044663 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b23f2560-5aa1-43b0-b0d9-674c6b0eb707-istio-envoy\") pod \"e2e-inf-b-6beee4-openshift-default-66d46f8786-2vth6\" (UID: \"b23f2560-5aa1-43b0-b0d9-674c6b0eb707\") " pod="openshift-ingress/e2e-inf-b-6beee4-openshift-default-66d46f8786-2vth6" Sep 10 13:29:52.045050 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:29:52.045008 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b23f2560-5aa1-43b0-b0d9-674c6b0eb707-istio-podinfo\") pod \"e2e-inf-b-6beee4-openshift-default-66d46f8786-2vth6\" (UID: \"b23f2560-5aa1-43b0-b0d9-674c6b0eb707\") " pod="openshift-ingress/e2e-inf-b-6beee4-openshift-default-66d46f8786-2vth6" Sep 10 13:29:52.057734 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:29:52.057632 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-htr6w\" (UniqueName: \"kubernetes.io/projected/b23f2560-5aa1-43b0-b0d9-674c6b0eb707-kube-api-access-htr6w\") pod \"e2e-inf-b-6beee4-openshift-default-66d46f8786-2vth6\" (UID: \"b23f2560-5aa1-43b0-b0d9-674c6b0eb707\") " pod="openshift-ingress/e2e-inf-b-6beee4-openshift-default-66d46f8786-2vth6" Sep 10 13:29:52.060306 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:29:52.060278 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b23f2560-5aa1-43b0-b0d9-674c6b0eb707-istio-token\") pod \"e2e-inf-b-6beee4-openshift-default-66d46f8786-2vth6\" (UID: \"b23f2560-5aa1-43b0-b0d9-674c6b0eb707\") " pod="openshift-ingress/e2e-inf-b-6beee4-openshift-default-66d46f8786-2vth6" Sep 10 13:29:52.070821 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:29:52.070791 2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ait-d2fb9c2a-846bdcb669-kh87w" Sep 10 13:29:52.125101 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:29:52.125054 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-6beee4-openshift-default-66d46f8786-2vth6" Sep 10 13:29:52.428640 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:29:52.428611 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-6beee4-openshift-default-66d46f8786-2vth6"] Sep 10 13:29:52.431090 ip-10-0-139-174 kubenswrapper[2557]: W0910 13:29:52.431064 2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb23f2560_5aa1_43b0_b0d9_674c6b0eb707.slice/crio-b03ee78ce8d64fb64faa2c8e36a213bf288ffcd00adfae9bb6d813f400108278 WatchSource:0}: Error finding container b03ee78ce8d64fb64faa2c8e36a213bf288ffcd00adfae9bb6d813f400108278: Status 404 returned error can't find the container with id b03ee78ce8d64fb64faa2c8e36a213bf288ffcd00adfae9bb6d813f400108278 Sep 10 13:29:52.434088 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:29:52.434054 2557 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892152Ki","pods":"250"} Sep 10 13:29:52.434173 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:29:52.434138 2557 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892152Ki","pods":"250"} Sep 10 13:29:52.434173 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:29:52.434167 2557 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892152Ki","pods":"250"} Sep 10 13:29:53.063634 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:29:53.063568 2557 patch_prober.go:28] interesting pod/e2e-ait-d2fb9c2a-openshift-default-689d78d8db-fxqrg container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.46:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Sep 10 13:29:53.064030 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:29:53.063659 2557 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ait-d2fb9c2a-openshift-default-689d78d8db-fxqrg" podUID="d605a297-fadf-4dc8-be1b-fc422c988207" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.46:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Sep 10 13:29:53.153123 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:29:53.153077 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-6beee4-openshift-default-66d46f8786-2vth6" event={"ID":"b23f2560-5aa1-43b0-b0d9-674c6b0eb707","Type":"ContainerStarted","Data":"843b18ac4f0ed976f156adbefa14a0a84d2086a03424b3842baa5ebbfdbb47a5"} Sep 10 13:29:53.153407 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:29:53.153141 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-6beee4-openshift-default-66d46f8786-2vth6" event={"ID":"b23f2560-5aa1-43b0-b0d9-674c6b0eb707","Type":"ContainerStarted","Data":"b03ee78ce8d64fb64faa2c8e36a213bf288ffcd00adfae9bb6d813f400108278"} Sep 10 13:29:53.180942 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:29:53.180877 2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-b-6beee4-openshift-default-66d46f8786-2vth6" podStartSLOduration=2.1808553330000002 podStartE2EDuration="2.180855333s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 13:29:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 13:29:53.178953444 +0000 UTC m=+753.485947884" watchObservedRunningTime="2026-09-10 13:29:53.180855333 +0000 UTC m=+753.487849748" Sep 10 13:29:54.125848 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:29:54.125811 2557 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-6beee4-openshift-default-66d46f8786-2vth6" Sep 10 13:29:54.131949 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:29:54.131927 2557 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-6beee4-openshift-default-66d46f8786-2vth6" Sep 10 13:29:54.160558 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:29:54.160530 2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-6beee4-openshift-default-66d46f8786-2vth6" Sep 10 13:29:54.161320 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:29:54.161299 2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-6beee4-openshift-default-66d46f8786-2vth6" Sep 10 13:29:55.168021 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:29:55.167987 2557 generic.go:356] "Generic (PLEG): container finished" podID="22c3a6a3-69e4-43db-8768-525668cb1928" containerID="1135f5e5d87cb02f53d9744e567fffc5f706cae4f2503f2886bb9597b97f0349" exitCode=0 Sep 10 13:29:55.168509 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:29:55.168074 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-autores-af59ca-openshift-default-887c6d76d-t4qb9" event={"ID":"22c3a6a3-69e4-43db-8768-525668cb1928","Type":"ContainerDied","Data":"1135f5e5d87cb02f53d9744e567fffc5f706cae4f2503f2886bb9597b97f0349"} Sep 10 13:29:55.168631 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:29:55.168611 2557 scope.go:117] "RemoveContainer" containerID="1135f5e5d87cb02f53d9744e567fffc5f706cae4f2503f2886bb9597b97f0349" Sep 10 13:29:55.169746 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:29:55.169715 2557 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892152Ki","pods":"250"} Sep 10 13:29:55.169860 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:29:55.169809 2557 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892152Ki","pods":"250"} Sep 10 13:29:55.169924 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:29:55.169856 2557 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892152Ki","pods":"250"} Sep 10 13:29:55.170066 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:29:55.170049 2557 generic.go:356] "Generic (PLEG): container finished" podID="d605a297-fadf-4dc8-be1b-fc422c988207" containerID="65fc3cadea708bae2c5dae204ea39db6e92f9d38eca8882749b4ca2ae93b83a3" exitCode=0 Sep 10 13:29:55.170132 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:29:55.170116 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-d2fb9c2a-openshift-default-689d78d8db-fxqrg" event={"ID":"d605a297-fadf-4dc8-be1b-fc422c988207","Type":"ContainerDied","Data":"65fc3cadea708bae2c5dae204ea39db6e92f9d38eca8882749b4ca2ae93b83a3"} Sep 10 13:29:55.170594 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:29:55.170578 2557 scope.go:117] "RemoveContainer" containerID="65fc3cadea708bae2c5dae204ea39db6e92f9d38eca8882749b4ca2ae93b83a3" Sep 10 13:29:55.171596 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:29:55.171570 2557 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892152Ki","pods":"250"} Sep 10 13:29:55.171680 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:29:55.171629 2557 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892152Ki","pods":"250"} Sep 10 13:29:55.171680 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:29:55.171662 2557 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892152Ki","pods":"250"} Sep 10 13:29:56.187170 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:29:56.187126 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-autores-af59ca-openshift-default-887c6d76d-t4qb9" event={"ID":"22c3a6a3-69e4-43db-8768-525668cb1928","Type":"ContainerStarted","Data":"caffa4bde3e2aecd434145033c8a8291c60306c41704f29f98bed8fb7fea89bb"} Sep 10 13:29:56.190996 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:29:56.190964 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-d2fb9c2a-openshift-default-689d78d8db-fxqrg" event={"ID":"d605a297-fadf-4dc8-be1b-fc422c988207","Type":"ContainerStarted","Data":"7350f70fe99f821b8ec96ec0b608ebbe7d16ce1525ed2cd506f100902e77e2e0"} Sep 10 13:29:56.438744 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:29:56.438638 2557 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-autores-af59ca-openshift-default-887c6d76d-t4qb9" Sep 10 13:29:56.438909 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:29:56.438766 2557 patch_prober.go:28] interesting pod/e2e-autores-af59ca-openshift-default-887c6d76d-t4qb9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.45:15021/healthz/ready\": dial tcp 10.134.0.45:15021: connect: connection refused" start-of-body= Sep 10 13:29:56.438909 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:29:56.438816 2557 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-autores-af59ca-openshift-default-887c6d76d-t4qb9" podUID="22c3a6a3-69e4-43db-8768-525668cb1928" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.45:15021/healthz/ready\": dial tcp 10.134.0.45:15021: connect: connection refused" Sep 10 13:29:56.635386 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:29:56.635350 2557 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-d2fb9c2a-openshift-default-689d78d8db-fxqrg" Sep 10 13:29:56.636244 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:29:56.636221 2557 patch_prober.go:28] interesting pod/e2e-ait-d2fb9c2a-openshift-default-689d78d8db-fxqrg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.46:15021/healthz/ready\": dial tcp 10.134.0.46:15021: connect: connection refused" start-of-body= Sep 10 13:29:56.636344 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:29:56.636276 2557 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-d2fb9c2a-openshift-default-689d78d8db-fxqrg" podUID="d605a297-fadf-4dc8-be1b-fc422c988207" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.46:15021/healthz/ready\": dial tcp 10.134.0.46:15021: connect: connection refused" Sep 10 13:29:57.438923 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:29:57.438889 2557 patch_prober.go:28] interesting pod/e2e-autores-af59ca-openshift-default-887c6d76d-t4qb9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.45:15021/healthz/ready\": dial tcp 10.134.0.45:15021: connect: connection refused" start-of-body= Sep 10 13:29:57.439287 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:29:57.438953 2557 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-autores-af59ca-openshift-default-887c6d76d-t4qb9" podUID="22c3a6a3-69e4-43db-8768-525668cb1928" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.45:15021/healthz/ready\": dial tcp 10.134.0.45:15021: connect: connection refused" Sep 10 13:29:57.636189 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:29:57.636144 2557 patch_prober.go:28] interesting pod/e2e-ait-d2fb9c2a-openshift-default-689d78d8db-fxqrg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.46:15021/healthz/ready\": dial tcp 10.134.0.46:15021: connect: connection refused" start-of-body= Sep 10 13:29:57.636400 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:29:57.636234 2557 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-d2fb9c2a-openshift-default-689d78d8db-fxqrg" podUID="d605a297-fadf-4dc8-be1b-fc422c988207" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.46:15021/healthz/ready\": dial tcp 10.134.0.46:15021: connect: connection refused" Sep 10 13:29:58.438786 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:29:58.438748 2557 patch_prober.go:28] interesting pod/e2e-autores-af59ca-openshift-default-887c6d76d-t4qb9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.45:15021/healthz/ready\": dial tcp 10.134.0.45:15021: connect: connection refused" start-of-body= Sep 10 13:29:58.438956 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:29:58.438799 2557 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-autores-af59ca-openshift-default-887c6d76d-t4qb9" podUID="22c3a6a3-69e4-43db-8768-525668cb1928" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.45:15021/healthz/ready\": dial tcp 10.134.0.45:15021: connect: connection refused" Sep 10 13:29:58.635743 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:29:58.635703 2557 patch_prober.go:28] interesting pod/e2e-ait-d2fb9c2a-openshift-default-689d78d8db-fxqrg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.46:15021/healthz/ready\": dial tcp 10.134.0.46:15021: connect: connection refused" start-of-body= Sep 10 13:29:58.635938 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:29:58.635767 2557 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-d2fb9c2a-openshift-default-689d78d8db-fxqrg" podUID="d605a297-fadf-4dc8-be1b-fc422c988207" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.46:15021/healthz/ready\": dial tcp 10.134.0.46:15021: connect: connection refused" Sep 10 13:29:59.439193 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:29:59.439154 2557 patch_prober.go:28] interesting pod/e2e-autores-af59ca-openshift-default-887c6d76d-t4qb9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.45:15021/healthz/ready\": dial tcp 10.134.0.45:15021: connect: connection refused" start-of-body= Sep 10 13:29:59.439665 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:29:59.439221 2557 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-autores-af59ca-openshift-default-887c6d76d-t4qb9" podUID="22c3a6a3-69e4-43db-8768-525668cb1928" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.45:15021/healthz/ready\": dial tcp 10.134.0.45:15021: connect: connection refused" Sep 10 13:29:59.638037 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:29:59.637998 2557 patch_prober.go:28] interesting pod/e2e-ait-d2fb9c2a-openshift-default-689d78d8db-fxqrg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.46:15021/healthz/ready\": dial tcp 10.134.0.46:15021: connect: connection refused" start-of-body= Sep 10 13:29:59.638203 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:29:59.638059 2557 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-d2fb9c2a-openshift-default-689d78d8db-fxqrg" podUID="d605a297-fadf-4dc8-be1b-fc422c988207" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.46:15021/healthz/ready\": dial tcp 10.134.0.46:15021: connect: connection refused" Sep 10 13:30:00.238964 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:30:00.238929 2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-autores-af59ca-29817450-4zl7r"] Sep 10 13:30:00.250998 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:30:00.250970 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-autores-af59ca-29817450-4zl7r" Sep 10 13:30:00.251786 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:30:00.251653 2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-39df6e-29817450-rhjv9"] Sep 10 13:30:00.254739 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:30:00.254713 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-wtblv\"" Sep 10 13:30:00.257187 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:30:00.257165 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-39df6e-29817450-rhjv9" Sep 10 13:30:00.269272 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:30:00.269242 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-autores-af59ca-29817450-4zl7r" Sep 10 13:30:00.279988 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:30:00.279873 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-autores-af59ca-29817450-4zl7r"] Sep 10 13:30:00.286055 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:30:00.286027 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-39df6e-29817450-rhjv9"] Sep 10 13:30:00.313452 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:30:00.312966 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-39df6e-29817450-rhjv9" Sep 10 13:30:00.438810 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:30:00.438773 2557 patch_prober.go:28] interesting pod/e2e-autores-af59ca-openshift-default-887c6d76d-t4qb9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.45:15021/healthz/ready\": dial tcp 10.134.0.45:15021: connect: connection refused" start-of-body= Sep 10 13:30:00.438974 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:30:00.438862 2557 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-autores-af59ca-openshift-default-887c6d76d-t4qb9" podUID="22c3a6a3-69e4-43db-8768-525668cb1928" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.45:15021/healthz/ready\": dial tcp 10.134.0.45:15021: connect: connection refused" Sep 10 13:30:00.635657 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:30:00.635621 2557 patch_prober.go:28] interesting pod/e2e-ait-d2fb9c2a-openshift-default-689d78d8db-fxqrg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.46:15021/healthz/ready\": dial tcp 10.134.0.46:15021: connect: connection refused" start-of-body= Sep 10 13:30:00.636126 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:30:00.635670 2557 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-d2fb9c2a-openshift-default-689d78d8db-fxqrg" podUID="d605a297-fadf-4dc8-be1b-fc422c988207" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.46:15021/healthz/ready\": dial tcp 10.134.0.46:15021: connect: connection refused" Sep 10 13:30:00.727021 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:30:00.726997 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-39df6e-29817450-rhjv9"] Sep 10 13:30:00.728198 ip-10-0-139-174 kubenswrapper[2557]: W0910 13:30:00.728175 2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod36e73f75_14e4_4fe3_81cf_546e6a2c10e1.slice/crio-498883bef02b8b18fe1b2b6af5eb6f74a969b510db43ad38fd7516b27b48f1ea WatchSource:0}: Error finding container 498883bef02b8b18fe1b2b6af5eb6f74a969b510db43ad38fd7516b27b48f1ea: Status 404 returned error can't find the container with id 498883bef02b8b18fe1b2b6af5eb6f74a969b510db43ad38fd7516b27b48f1ea Sep 10 13:30:00.730207 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:30:00.730187 2557 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Sep 10 13:30:00.738110 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:30:00.738089 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-autores-af59ca-29817450-4zl7r"] Sep 10 13:30:00.738950 ip-10-0-139-174 kubenswrapper[2557]: W0910 13:30:00.738931 2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc77626f4_aa4b_49b9_8a28_fb070afa5c26.slice/crio-034e08c0a0fa433f2dc72b1847e0079dd2ff8107d2dbd0b786d8621b36507d52 WatchSource:0}: Error finding container 034e08c0a0fa433f2dc72b1847e0079dd2ff8107d2dbd0b786d8621b36507d52: Status 404 returned error can't find the container with id 034e08c0a0fa433f2dc72b1847e0079dd2ff8107d2dbd0b786d8621b36507d52 Sep 10 13:30:01.230216 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:30:01.230174 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-39df6e-29817450-rhjv9" event={"ID":"36e73f75-14e4-4fe3-81cf-546e6a2c10e1","Type":"ContainerStarted","Data":"498883bef02b8b18fe1b2b6af5eb6f74a969b510db43ad38fd7516b27b48f1ea"} Sep 10 13:30:01.232227 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:30:01.232197 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-autores-af59ca-29817450-4zl7r" event={"ID":"c77626f4-aa4b-49b9-8a28-fb070afa5c26","Type":"ContainerStarted","Data":"034e08c0a0fa433f2dc72b1847e0079dd2ff8107d2dbd0b786d8621b36507d52"} Sep 10 13:30:01.645555 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:30:01.645470 2557 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-d2fb9c2a-openshift-default-689d78d8db-fxqrg" Sep 10 13:30:01.646865 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:30:01.646836 2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-d2fb9c2a-openshift-default-689d78d8db-fxqrg" Sep 10 13:30:01.649538 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:30:01.649514 2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-d2fb9c2a-openshift-default-689d78d8db-fxqrg" Sep 10 13:30:01.732167 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:30:01.732135 2557 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-autores-af59ca-openshift-default-887c6d76d-t4qb9" Sep 10 13:30:01.732849 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:30:01.732801 2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-autores-af59ca-openshift-default-887c6d76d-t4qb9" Sep 10 13:30:01.734088 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:30:01.734047 2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-autores-af59ca-openshift-default-887c6d76d-t4qb9" Sep 10 13:30:05.287293 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:30:05.287258 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-39df6e-29817450-rhjv9" event={"ID":"36e73f75-14e4-4fe3-81cf-546e6a2c10e1","Type":"ContainerStarted","Data":"bea50f533924e113fb4d6777c58e74f6f2d1371693833998bfb29b2bc9f387eb"} Sep 10 13:30:05.288930 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:30:05.288908 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-autores-af59ca-29817450-4zl7r" event={"ID":"c77626f4-aa4b-49b9-8a28-fb070afa5c26","Type":"ContainerStarted","Data":"2ec382f4e6e076dd64cdbf0404509c3767f7225af9f9287ba2d9ed797574fca7"} Sep 10 13:30:05.309498 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:30:05.309458 2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-39df6e-29817450-rhjv9" podStartSLOduration=1.8859994279999999 podStartE2EDuration="5.309444744s" totalImagesPullingTime="3.423445316s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 13:30:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 13:30:05.307007249 +0000 UTC m=+765.614001663" watchObservedRunningTime="2026-09-10 13:30:05.309444744 +0000 UTC m=+765.616439155" Sep 10 13:30:05.336379 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:30:05.336338 2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-autores-af59ca-29817450-4zl7r" podStartSLOduration=1.925982827 podStartE2EDuration="5.33632374s" totalImagesPullingTime="3.410340913s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 13:30:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 13:30:05.33288995 +0000 UTC m=+765.639884363" watchObservedRunningTime="2026-09-10 13:30:05.33632374 +0000 UTC m=+765.643318154" Sep 10 13:30:06.297236 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:30:06.297197 2557 generic.go:356] "Generic (PLEG): container finished" podID="c77626f4-aa4b-49b9-8a28-fb070afa5c26" containerID="2ec382f4e6e076dd64cdbf0404509c3767f7225af9f9287ba2d9ed797574fca7" exitCode=0 Sep 10 13:30:06.297720 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:30:06.297284 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-autores-af59ca-29817450-4zl7r" event={"ID":"c77626f4-aa4b-49b9-8a28-fb070afa5c26","Type":"ContainerDied","Data":"2ec382f4e6e076dd64cdbf0404509c3767f7225af9f9287ba2d9ed797574fca7"} Sep 10 13:30:06.299034 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:30:06.299006 2557 generic.go:356] "Generic (PLEG): container finished" podID="36e73f75-14e4-4fe3-81cf-546e6a2c10e1" containerID="bea50f533924e113fb4d6777c58e74f6f2d1371693833998bfb29b2bc9f387eb" exitCode=0 Sep 10 13:30:06.299134 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:30:06.299054 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-39df6e-29817450-rhjv9" event={"ID":"36e73f75-14e4-4fe3-81cf-546e6a2c10e1","Type":"ContainerDied","Data":"bea50f533924e113fb4d6777c58e74f6f2d1371693833998bfb29b2bc9f387eb"} Sep 10 13:30:07.490650 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:30:07.490622 2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-39df6e-29817450-rhjv9" Sep 10 13:30:07.493951 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:30:07.493928 2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-autores-af59ca-29817450-4zl7r" Sep 10 13:30:08.312704 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:30:08.312669 2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-39df6e-29817450-rhjv9" Sep 10 13:30:08.312877 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:30:08.312672 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-39df6e-29817450-rhjv9" event={"ID":"36e73f75-14e4-4fe3-81cf-546e6a2c10e1","Type":"ContainerDied","Data":"498883bef02b8b18fe1b2b6af5eb6f74a969b510db43ad38fd7516b27b48f1ea"} Sep 10 13:30:08.312877 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:30:08.312826 2557 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="498883bef02b8b18fe1b2b6af5eb6f74a969b510db43ad38fd7516b27b48f1ea" Sep 10 13:30:08.314224 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:30:08.314200 2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-autores-af59ca-29817450-4zl7r" Sep 10 13:30:08.314331 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:30:08.314222 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-autores-af59ca-29817450-4zl7r" event={"ID":"c77626f4-aa4b-49b9-8a28-fb070afa5c26","Type":"ContainerDied","Data":"034e08c0a0fa433f2dc72b1847e0079dd2ff8107d2dbd0b786d8621b36507d52"} Sep 10 13:30:08.314331 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:30:08.314263 2557 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="034e08c0a0fa433f2dc72b1847e0079dd2ff8107d2dbd0b786d8621b36507d52" Sep 10 13:30:09.946080 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:30:09.946043 2557 patch_prober.go:28] interesting pod/e2e-inf-b-6beee4-openshift-default-66d46f8786-2vth6 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.48:15021/healthz/ready\": read tcp 10.134.0.2:38474->10.134.0.48:15021: read: connection reset by peer" start-of-body= Sep 10 13:30:09.946559 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:30:09.946122 2557 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-b-6beee4-openshift-default-66d46f8786-2vth6" podUID="b23f2560-5aa1-43b0-b0d9-674c6b0eb707" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.48:15021/healthz/ready\": read tcp 10.134.0.2:38474->10.134.0.48:15021: read: connection reset by peer" Sep 10 13:30:10.327995 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:30:10.327964 2557 generic.go:356] "Generic (PLEG): container finished" podID="b23f2560-5aa1-43b0-b0d9-674c6b0eb707" containerID="843b18ac4f0ed976f156adbefa14a0a84d2086a03424b3842baa5ebbfdbb47a5" exitCode=0 Sep 10 13:30:10.328169 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:30:10.328012 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-6beee4-openshift-default-66d46f8786-2vth6" event={"ID":"b23f2560-5aa1-43b0-b0d9-674c6b0eb707","Type":"ContainerDied","Data":"843b18ac4f0ed976f156adbefa14a0a84d2086a03424b3842baa5ebbfdbb47a5"} Sep 10 13:30:10.328461 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:30:10.328411 2557 scope.go:117] "RemoveContainer" containerID="843b18ac4f0ed976f156adbefa14a0a84d2086a03424b3842baa5ebbfdbb47a5" Sep 10 13:30:10.329639 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:30:10.329605 2557 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892152Ki","pods":"250"} Sep 10 13:30:10.329764 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:30:10.329676 2557 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892152Ki","pods":"250"} Sep 10 13:30:10.329764 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:30:10.329718 2557 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892152Ki","pods":"250"} Sep 10 13:30:11.338307 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:30:11.338270 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-6beee4-openshift-default-66d46f8786-2vth6" event={"ID":"b23f2560-5aa1-43b0-b0d9-674c6b0eb707","Type":"ContainerStarted","Data":"9a288f766cf74d1e24c89ffb2db04b173dc09e04adb328b49e8050ffe16f4b38"} Sep 10 13:30:12.125570 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:30:12.125514 2557 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-6beee4-openshift-default-66d46f8786-2vth6" Sep 10 13:30:12.125867 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:30:12.125841 2557 patch_prober.go:28] interesting pod/e2e-inf-b-6beee4-openshift-default-66d46f8786-2vth6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.48:15021/healthz/ready\": dial tcp 10.134.0.48:15021: connect: connection refused" start-of-body= Sep 10 13:30:12.125981 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:30:12.125902 2557 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-6beee4-openshift-default-66d46f8786-2vth6" podUID="b23f2560-5aa1-43b0-b0d9-674c6b0eb707" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.48:15021/healthz/ready\": dial tcp 10.134.0.48:15021: connect: connection refused" Sep 10 13:30:13.126335 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:30:13.126298 2557 patch_prober.go:28] interesting pod/e2e-inf-b-6beee4-openshift-default-66d46f8786-2vth6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.48:15021/healthz/ready\": dial tcp 10.134.0.48:15021: connect: connection refused" start-of-body= Sep 10 13:30:13.126802 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:30:13.126367 2557 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-6beee4-openshift-default-66d46f8786-2vth6" podUID="b23f2560-5aa1-43b0-b0d9-674c6b0eb707" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.48:15021/healthz/ready\": dial tcp 10.134.0.48:15021: connect: connection refused" Sep 10 13:30:14.125876 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:30:14.125841 2557 patch_prober.go:28] interesting pod/e2e-inf-b-6beee4-openshift-default-66d46f8786-2vth6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.48:15021/healthz/ready\": dial tcp 10.134.0.48:15021: connect: connection refused" start-of-body= Sep 10 13:30:14.126055 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:30:14.125925 2557 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-6beee4-openshift-default-66d46f8786-2vth6" podUID="b23f2560-5aa1-43b0-b0d9-674c6b0eb707" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.48:15021/healthz/ready\": dial tcp 10.134.0.48:15021: connect: connection refused" Sep 10 13:30:15.126213 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:30:15.126171 2557 patch_prober.go:28] interesting pod/e2e-inf-b-6beee4-openshift-default-66d46f8786-2vth6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.48:15021/healthz/ready\": dial tcp 10.134.0.48:15021: connect: connection refused" start-of-body= Sep 10 13:30:15.126683 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:30:15.126253 2557 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-6beee4-openshift-default-66d46f8786-2vth6" podUID="b23f2560-5aa1-43b0-b0d9-674c6b0eb707" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.48:15021/healthz/ready\": dial tcp 10.134.0.48:15021: connect: connection refused" Sep 10 13:30:16.126528 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:30:16.126490 2557 patch_prober.go:28] interesting pod/e2e-inf-b-6beee4-openshift-default-66d46f8786-2vth6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.48:15021/healthz/ready\": dial tcp 10.134.0.48:15021: connect: connection refused" start-of-body= Sep 10 13:30:16.126924 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:30:16.126557 2557 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-6beee4-openshift-default-66d46f8786-2vth6" podUID="b23f2560-5aa1-43b0-b0d9-674c6b0eb707" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.48:15021/healthz/ready\": dial tcp 10.134.0.48:15021: connect: connection refused" Sep 10 13:30:17.129837 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:30:17.129805 2557 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-6beee4-openshift-default-66d46f8786-2vth6" Sep 10 13:30:17.130284 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:30:17.130104 2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-6beee4-openshift-default-66d46f8786-2vth6" Sep 10 13:30:17.130878 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:30:17.130855 2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-6beee4-openshift-default-66d46f8786-2vth6" Sep 10 13:30:19.085087 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:30:19.085050 2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-d2fb9c2a-846bdcb669-kh87w"] Sep 10 13:30:19.085486 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:30:19.085322 2557 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ait-d2fb9c2a-846bdcb669-kh87w" podUID="1883269d-5793-47cf-b93a-35658a57e489" containerName="payload-processing" containerID="cri-o://fa3967ca67ae7375d11b8257ed78e9df9e561c87fe613032b4185bc9d0c4d506" gracePeriod=30 Sep 10 13:30:19.371787 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:30:19.371754 2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-d2fb9c2a-846bdcb669-kh87w" Sep 10 13:30:19.403527 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:30:19.403491 2557 generic.go:356] "Generic (PLEG): container finished" podID="1883269d-5793-47cf-b93a-35658a57e489" containerID="fa3967ca67ae7375d11b8257ed78e9df9e561c87fe613032b4185bc9d0c4d506" exitCode=0 Sep 10 13:30:19.403710 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:30:19.403531 2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-d2fb9c2a-846bdcb669-kh87w" Sep 10 13:30:19.403710 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:30:19.403588 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-d2fb9c2a-846bdcb669-kh87w" event={"ID":"1883269d-5793-47cf-b93a-35658a57e489","Type":"ContainerDied","Data":"fa3967ca67ae7375d11b8257ed78e9df9e561c87fe613032b4185bc9d0c4d506"} Sep 10 13:30:19.403710 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:30:19.403642 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-d2fb9c2a-846bdcb669-kh87w" event={"ID":"1883269d-5793-47cf-b93a-35658a57e489","Type":"ContainerDied","Data":"9b6e5b39772ab78f5ec84e79b245ad6fd7c0b68929847a3a578505d68c75c7e3"} Sep 10 13:30:19.403710 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:30:19.403674 2557 scope.go:117] "RemoveContainer" containerID="fa3967ca67ae7375d11b8257ed78e9df9e561c87fe613032b4185bc9d0c4d506" Sep 10 13:30:19.423034 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:30:19.422991 2557 scope.go:117] "RemoveContainer" containerID="fa3967ca67ae7375d11b8257ed78e9df9e561c87fe613032b4185bc9d0c4d506" Sep 10 13:30:19.423524 ip-10-0-139-174 kubenswrapper[2557]: E0910 13:30:19.423492 2557 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fa3967ca67ae7375d11b8257ed78e9df9e561c87fe613032b4185bc9d0c4d506\": container with ID starting with fa3967ca67ae7375d11b8257ed78e9df9e561c87fe613032b4185bc9d0c4d506 not found: ID does not exist" containerID="fa3967ca67ae7375d11b8257ed78e9df9e561c87fe613032b4185bc9d0c4d506" Sep 10 13:30:19.423708 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:30:19.423675 2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fa3967ca67ae7375d11b8257ed78e9df9e561c87fe613032b4185bc9d0c4d506"} err="failed to get container status \"fa3967ca67ae7375d11b8257ed78e9df9e561c87fe613032b4185bc9d0c4d506\": rpc error: code = NotFound desc = could not find container \"fa3967ca67ae7375d11b8257ed78e9df9e561c87fe613032b4185bc9d0c4d506\": container with ID starting with fa3967ca67ae7375d11b8257ed78e9df9e561c87fe613032b4185bc9d0c4d506 not found: ID does not exist" Sep 10 13:30:19.478624 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:30:19.478594 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1883269d-5793-47cf-b93a-35658a57e489-plugins-config-volume\") pod \"1883269d-5793-47cf-b93a-35658a57e489\" (UID: \"1883269d-5793-47cf-b93a-35658a57e489\") " Sep 10 13:30:19.478819 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:30:19.478645 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jhmt8\" (UniqueName: \"kubernetes.io/projected/1883269d-5793-47cf-b93a-35658a57e489-kube-api-access-jhmt8\") pod \"1883269d-5793-47cf-b93a-35658a57e489\" (UID: \"1883269d-5793-47cf-b93a-35658a57e489\") " Sep 10 13:30:19.479133 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:30:19.479101 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1883269d-5793-47cf-b93a-35658a57e489-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "1883269d-5793-47cf-b93a-35658a57e489" (UID: "1883269d-5793-47cf-b93a-35658a57e489"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 10 13:30:19.482892 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:30:19.482292 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1883269d-5793-47cf-b93a-35658a57e489-kube-api-access-jhmt8" (OuterVolumeSpecName: "kube-api-access-jhmt8") pod "1883269d-5793-47cf-b93a-35658a57e489" (UID: "1883269d-5793-47cf-b93a-35658a57e489"). InnerVolumeSpecName "kube-api-access-jhmt8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 10 13:30:19.486591 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:30:19.486559 2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-d2fb9c2a-openshift-default-689d78d8db-fxqrg"] Sep 10 13:30:19.487545 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:30:19.486831 2557 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-ait-d2fb9c2a-openshift-default-689d78d8db-fxqrg" podUID="d605a297-fadf-4dc8-be1b-fc422c988207" containerName="istio-proxy" containerID="cri-o://7350f70fe99f821b8ec96ec0b608ebbe7d16ce1525ed2cd506f100902e77e2e0" gracePeriod=30 Sep 10 13:30:19.580119 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:30:19.580086 2557 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1883269d-5793-47cf-b93a-35658a57e489-plugins-config-volume\") on node \"ip-10-0-139-174.ec2.internal\" DevicePath \"\"" Sep 10 13:30:19.580119 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:30:19.580113 2557 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jhmt8\" (UniqueName: \"kubernetes.io/projected/1883269d-5793-47cf-b93a-35658a57e489-kube-api-access-jhmt8\") on node \"ip-10-0-139-174.ec2.internal\" DevicePath \"\"" Sep 10 13:30:19.732896 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:30:19.732860 2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-d2fb9c2a-846bdcb669-kh87w"] Sep 10 13:30:19.741835 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:30:19.741804 2557 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-d2fb9c2a-846bdcb669-kh87w"] Sep 10 13:30:20.220187 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:30:20.220158 2557 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1883269d-5793-47cf-b93a-35658a57e489" path="/var/lib/kubelet/pods/1883269d-5793-47cf-b93a-35658a57e489/volumes" Sep 10 13:30:24.766174 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:30:24.766107 2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-d2fb9c2a-openshift-default-689d78d8db-fxqrg" Sep 10 13:30:24.930668 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:30:24.930628 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d605a297-fadf-4dc8-be1b-fc422c988207-istiod-ca-cert\") pod \"d605a297-fadf-4dc8-be1b-fc422c988207\" (UID: \"d605a297-fadf-4dc8-be1b-fc422c988207\") " Sep 10 13:30:24.930852 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:30:24.930681 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d605a297-fadf-4dc8-be1b-fc422c988207-workload-socket\") pod \"d605a297-fadf-4dc8-be1b-fc422c988207\" (UID: \"d605a297-fadf-4dc8-be1b-fc422c988207\") " Sep 10 13:30:24.930852 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:30:24.930737 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d605a297-fadf-4dc8-be1b-fc422c988207-istio-envoy\") pod \"d605a297-fadf-4dc8-be1b-fc422c988207\" (UID: \"d605a297-fadf-4dc8-be1b-fc422c988207\") " Sep 10 13:30:24.930852 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:30:24.930760 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d605a297-fadf-4dc8-be1b-fc422c988207-istio-podinfo\") pod \"d605a297-fadf-4dc8-be1b-fc422c988207\" (UID: \"d605a297-fadf-4dc8-be1b-fc422c988207\") " Sep 10 13:30:24.930852 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:30:24.930791 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-plw22\" (UniqueName: \"kubernetes.io/projected/d605a297-fadf-4dc8-be1b-fc422c988207-kube-api-access-plw22\") pod \"d605a297-fadf-4dc8-be1b-fc422c988207\" (UID: \"d605a297-fadf-4dc8-be1b-fc422c988207\") " Sep 10 13:30:24.930852 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:30:24.930838 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d605a297-fadf-4dc8-be1b-fc422c988207-istio-token\") pod \"d605a297-fadf-4dc8-be1b-fc422c988207\" (UID: \"d605a297-fadf-4dc8-be1b-fc422c988207\") " Sep 10 13:30:24.931190 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:30:24.930873 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d605a297-fadf-4dc8-be1b-fc422c988207-workload-certs\") pod \"d605a297-fadf-4dc8-be1b-fc422c988207\" (UID: \"d605a297-fadf-4dc8-be1b-fc422c988207\") " Sep 10 13:30:24.931190 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:30:24.930955 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d605a297-fadf-4dc8-be1b-fc422c988207-istio-data\") pod \"d605a297-fadf-4dc8-be1b-fc422c988207\" (UID: \"d605a297-fadf-4dc8-be1b-fc422c988207\") " Sep 10 13:30:24.931190 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:30:24.930982 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d605a297-fadf-4dc8-be1b-fc422c988207-credential-socket\") pod \"d605a297-fadf-4dc8-be1b-fc422c988207\" (UID: \"d605a297-fadf-4dc8-be1b-fc422c988207\") " Sep 10 13:30:24.931190 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:30:24.931092 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d605a297-fadf-4dc8-be1b-fc422c988207-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "d605a297-fadf-4dc8-be1b-fc422c988207" (UID: "d605a297-fadf-4dc8-be1b-fc422c988207"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 13:30:24.931190 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:30:24.931025 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d605a297-fadf-4dc8-be1b-fc422c988207-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "d605a297-fadf-4dc8-be1b-fc422c988207" (UID: "d605a297-fadf-4dc8-be1b-fc422c988207"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 10 13:30:24.931496 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:30:24.931271 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d605a297-fadf-4dc8-be1b-fc422c988207-istio-data" (OuterVolumeSpecName: "istio-data") pod "d605a297-fadf-4dc8-be1b-fc422c988207" (UID: "d605a297-fadf-4dc8-be1b-fc422c988207"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 13:30:24.931496 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:30:24.931327 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d605a297-fadf-4dc8-be1b-fc422c988207-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "d605a297-fadf-4dc8-be1b-fc422c988207" (UID: "d605a297-fadf-4dc8-be1b-fc422c988207"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 13:30:24.931496 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:30:24.931453 2557 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d605a297-fadf-4dc8-be1b-fc422c988207-istio-data\") on node \"ip-10-0-139-174.ec2.internal\" DevicePath \"\"" Sep 10 13:30:24.931496 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:30:24.931476 2557 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d605a297-fadf-4dc8-be1b-fc422c988207-credential-socket\") on node \"ip-10-0-139-174.ec2.internal\" DevicePath \"\"" Sep 10 13:30:24.931496 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:30:24.931489 2557 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d605a297-fadf-4dc8-be1b-fc422c988207-istiod-ca-cert\") on node \"ip-10-0-139-174.ec2.internal\" DevicePath \"\"" Sep 10 13:30:24.931928 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:30:24.931502 2557 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d605a297-fadf-4dc8-be1b-fc422c988207-workload-socket\") on node \"ip-10-0-139-174.ec2.internal\" DevicePath \"\"" Sep 10 13:30:24.931928 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:30:24.931621 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d605a297-fadf-4dc8-be1b-fc422c988207-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "d605a297-fadf-4dc8-be1b-fc422c988207" (UID: "d605a297-fadf-4dc8-be1b-fc422c988207"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 13:30:24.933284 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:30:24.933258 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d605a297-fadf-4dc8-be1b-fc422c988207-kube-api-access-plw22" (OuterVolumeSpecName: "kube-api-access-plw22") pod "d605a297-fadf-4dc8-be1b-fc422c988207" (UID: "d605a297-fadf-4dc8-be1b-fc422c988207"). InnerVolumeSpecName "kube-api-access-plw22". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 10 13:30:24.933483 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:30:24.933463 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d605a297-fadf-4dc8-be1b-fc422c988207-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "d605a297-fadf-4dc8-be1b-fc422c988207" (UID: "d605a297-fadf-4dc8-be1b-fc422c988207"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 13:30:24.933758 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:30:24.933737 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d605a297-fadf-4dc8-be1b-fc422c988207-istio-token" (OuterVolumeSpecName: "istio-token") pod "d605a297-fadf-4dc8-be1b-fc422c988207" (UID: "d605a297-fadf-4dc8-be1b-fc422c988207"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 10 13:30:24.933836 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:30:24.933789 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/d605a297-fadf-4dc8-be1b-fc422c988207-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "d605a297-fadf-4dc8-be1b-fc422c988207" (UID: "d605a297-fadf-4dc8-be1b-fc422c988207"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Sep 10 13:30:25.032840 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:30:25.032757 2557 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d605a297-fadf-4dc8-be1b-fc422c988207-istio-envoy\") on node \"ip-10-0-139-174.ec2.internal\" DevicePath \"\"" Sep 10 13:30:25.032840 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:30:25.032783 2557 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d605a297-fadf-4dc8-be1b-fc422c988207-istio-podinfo\") on node \"ip-10-0-139-174.ec2.internal\" DevicePath \"\"" Sep 10 13:30:25.032840 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:30:25.032792 2557 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-plw22\" (UniqueName: \"kubernetes.io/projected/d605a297-fadf-4dc8-be1b-fc422c988207-kube-api-access-plw22\") on node \"ip-10-0-139-174.ec2.internal\" DevicePath \"\"" Sep 10 13:30:25.032840 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:30:25.032802 2557 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d605a297-fadf-4dc8-be1b-fc422c988207-istio-token\") on node \"ip-10-0-139-174.ec2.internal\" DevicePath \"\"" Sep 10 13:30:25.032840 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:30:25.032810 2557 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d605a297-fadf-4dc8-be1b-fc422c988207-workload-certs\") on node \"ip-10-0-139-174.ec2.internal\" DevicePath \"\"" Sep 10 13:30:25.442863 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:30:25.442766 2557 generic.go:356] "Generic (PLEG): container finished" podID="d605a297-fadf-4dc8-be1b-fc422c988207" containerID="7350f70fe99f821b8ec96ec0b608ebbe7d16ce1525ed2cd506f100902e77e2e0" exitCode=0 Sep 10 13:30:25.442863 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:30:25.442817 2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-d2fb9c2a-openshift-default-689d78d8db-fxqrg" Sep 10 13:30:25.443043 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:30:25.442808 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-d2fb9c2a-openshift-default-689d78d8db-fxqrg" event={"ID":"d605a297-fadf-4dc8-be1b-fc422c988207","Type":"ContainerDied","Data":"7350f70fe99f821b8ec96ec0b608ebbe7d16ce1525ed2cd506f100902e77e2e0"} Sep 10 13:30:25.443043 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:30:25.442954 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-d2fb9c2a-openshift-default-689d78d8db-fxqrg" event={"ID":"d605a297-fadf-4dc8-be1b-fc422c988207","Type":"ContainerDied","Data":"1a2d778c40f97870d4cd5ab42c30395ed214c937264b5996f8dc6a9d5f10a8e5"} Sep 10 13:30:25.443043 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:30:25.442982 2557 scope.go:117] "RemoveContainer" containerID="7350f70fe99f821b8ec96ec0b608ebbe7d16ce1525ed2cd506f100902e77e2e0" Sep 10 13:30:25.453552 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:30:25.453535 2557 scope.go:117] "RemoveContainer" containerID="65fc3cadea708bae2c5dae204ea39db6e92f9d38eca8882749b4ca2ae93b83a3" Sep 10 13:30:25.462185 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:30:25.462167 2557 scope.go:117] "RemoveContainer" containerID="7350f70fe99f821b8ec96ec0b608ebbe7d16ce1525ed2cd506f100902e77e2e0" Sep 10 13:30:25.462503 ip-10-0-139-174 kubenswrapper[2557]: E0910 13:30:25.462483 2557 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7350f70fe99f821b8ec96ec0b608ebbe7d16ce1525ed2cd506f100902e77e2e0\": container with ID starting with 7350f70fe99f821b8ec96ec0b608ebbe7d16ce1525ed2cd506f100902e77e2e0 not found: ID does not exist" containerID="7350f70fe99f821b8ec96ec0b608ebbe7d16ce1525ed2cd506f100902e77e2e0" Sep 10 13:30:25.462558 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:30:25.462513 2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7350f70fe99f821b8ec96ec0b608ebbe7d16ce1525ed2cd506f100902e77e2e0"} err="failed to get container status \"7350f70fe99f821b8ec96ec0b608ebbe7d16ce1525ed2cd506f100902e77e2e0\": rpc error: code = NotFound desc = could not find container \"7350f70fe99f821b8ec96ec0b608ebbe7d16ce1525ed2cd506f100902e77e2e0\": container with ID starting with 7350f70fe99f821b8ec96ec0b608ebbe7d16ce1525ed2cd506f100902e77e2e0 not found: ID does not exist" Sep 10 13:30:25.462558 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:30:25.462531 2557 scope.go:117] "RemoveContainer" containerID="65fc3cadea708bae2c5dae204ea39db6e92f9d38eca8882749b4ca2ae93b83a3" Sep 10 13:30:25.462788 ip-10-0-139-174 kubenswrapper[2557]: E0910 13:30:25.462771 2557 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"65fc3cadea708bae2c5dae204ea39db6e92f9d38eca8882749b4ca2ae93b83a3\": container with ID starting with 65fc3cadea708bae2c5dae204ea39db6e92f9d38eca8882749b4ca2ae93b83a3 not found: ID does not exist" containerID="65fc3cadea708bae2c5dae204ea39db6e92f9d38eca8882749b4ca2ae93b83a3" Sep 10 13:30:25.462834 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:30:25.462795 2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"65fc3cadea708bae2c5dae204ea39db6e92f9d38eca8882749b4ca2ae93b83a3"} err="failed to get container status \"65fc3cadea708bae2c5dae204ea39db6e92f9d38eca8882749b4ca2ae93b83a3\": rpc error: code = NotFound desc = could not find container \"65fc3cadea708bae2c5dae204ea39db6e92f9d38eca8882749b4ca2ae93b83a3\": container with ID starting with 65fc3cadea708bae2c5dae204ea39db6e92f9d38eca8882749b4ca2ae93b83a3 not found: ID does not exist" Sep 10 13:30:25.473384 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:30:25.473363 2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-d2fb9c2a-openshift-default-689d78d8db-fxqrg"] Sep 10 13:30:25.478368 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:30:25.478348 2557 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ait-d2fb9c2a-openshift-default-689d78d8db-fxqrg"] Sep 10 13:30:26.220705 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:30:26.220675 2557 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d605a297-fadf-4dc8-be1b-fc422c988207" path="/var/lib/kubelet/pods/d605a297-fadf-4dc8-be1b-fc422c988207/volumes" Sep 10 13:30:28.477060 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:30:28.477027 2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-7b43a88f-855895d994-7r7zc"] Sep 10 13:30:28.477457 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:30:28.477444 2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d605a297-fadf-4dc8-be1b-fc422c988207" containerName="istio-proxy" Sep 10 13:30:28.477509 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:30:28.477458 2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="d605a297-fadf-4dc8-be1b-fc422c988207" containerName="istio-proxy" Sep 10 13:30:28.477509 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:30:28.477470 2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c77626f4-aa4b-49b9-8a28-fb070afa5c26" containerName="cleanup" Sep 10 13:30:28.477509 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:30:28.477477 2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="c77626f4-aa4b-49b9-8a28-fb070afa5c26" containerName="cleanup" Sep 10 13:30:28.477509 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:30:28.477484 2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d605a297-fadf-4dc8-be1b-fc422c988207" containerName="istio-proxy" Sep 10 13:30:28.477509 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:30:28.477489 2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="d605a297-fadf-4dc8-be1b-fc422c988207" containerName="istio-proxy" Sep 10 13:30:28.477509 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:30:28.477496 2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="36e73f75-14e4-4fe3-81cf-546e6a2c10e1" containerName="cleanup" Sep 10 13:30:28.477509 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:30:28.477501 2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="36e73f75-14e4-4fe3-81cf-546e6a2c10e1" containerName="cleanup" Sep 10 13:30:28.477509 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:30:28.477507 2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1883269d-5793-47cf-b93a-35658a57e489" containerName="payload-processing" Sep 10 13:30:28.477509 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:30:28.477512 2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="1883269d-5793-47cf-b93a-35658a57e489" containerName="payload-processing" Sep 10 13:30:28.477784 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:30:28.477582 2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="d605a297-fadf-4dc8-be1b-fc422c988207" containerName="istio-proxy" Sep 10 13:30:28.477784 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:30:28.477589 2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="1883269d-5793-47cf-b93a-35658a57e489" containerName="payload-processing" Sep 10 13:30:28.477784 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:30:28.477597 2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="c77626f4-aa4b-49b9-8a28-fb070afa5c26" containerName="cleanup" Sep 10 13:30:28.477784 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:30:28.477605 2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="36e73f75-14e4-4fe3-81cf-546e6a2c10e1" containerName="cleanup" Sep 10 13:30:28.484451 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:30:28.482472 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-7b43a88f-855895d994-7r7zc" Sep 10 13:30:28.486350 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:30:28.485001 2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-migrate-7b43a88f\"" Sep 10 13:30:28.486350 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:30:28.485221 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-migrate-7b43a88f-dockercfg-7bv8r\"" Sep 10 13:30:28.498745 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:30:28.498721 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-7b43a88f-855895d994-7r7zc"] Sep 10 13:30:28.509391 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:30:28.509354 2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-7b43a88f-6fdf74449d-99d59"] Sep 10 13:30:28.510125 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:30:28.510106 2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="d605a297-fadf-4dc8-be1b-fc422c988207" containerName="istio-proxy" Sep 10 13:30:28.513590 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:30:28.513574 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-7b43a88f-6fdf74449d-99d59" Sep 10 13:30:28.524345 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:30:28.524324 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-7b43a88f-6fdf74449d-99d59"] Sep 10 13:30:28.564961 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:30:28.564930 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5kbtc\" (UniqueName: \"kubernetes.io/projected/1cce2cdc-ebd5-43f5-bb06-c0817da4e510-kube-api-access-5kbtc\") pod \"payload-pre-processing-e2e-migrate-7b43a88f-855895d994-7r7zc\" (UID: \"1cce2cdc-ebd5-43f5-bb06-c0817da4e510\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-7b43a88f-855895d994-7r7zc" Sep 10 13:30:28.565100 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:30:28.564994 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1cce2cdc-ebd5-43f5-bb06-c0817da4e510-plugins-config-volume\") pod \"payload-pre-processing-e2e-migrate-7b43a88f-855895d994-7r7zc\" (UID: \"1cce2cdc-ebd5-43f5-bb06-c0817da4e510\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-7b43a88f-855895d994-7r7zc" Sep 10 13:30:28.665764 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:30:28.665722 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zrpb2\" (UniqueName: \"kubernetes.io/projected/090bc9d2-7f8c-41b3-abd1-4b608891dba6-kube-api-access-zrpb2\") pod \"payload-processing-e2e-migrate-7b43a88f-6fdf74449d-99d59\" (UID: \"090bc9d2-7f8c-41b3-abd1-4b608891dba6\") " pod="openshift-ingress/payload-processing-e2e-migrate-7b43a88f-6fdf74449d-99d59" Sep 10 13:30:28.665964 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:30:28.665806 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5kbtc\" (UniqueName: \"kubernetes.io/projected/1cce2cdc-ebd5-43f5-bb06-c0817da4e510-kube-api-access-5kbtc\") pod \"payload-pre-processing-e2e-migrate-7b43a88f-855895d994-7r7zc\" (UID: \"1cce2cdc-ebd5-43f5-bb06-c0817da4e510\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-7b43a88f-855895d994-7r7zc" Sep 10 13:30:28.665964 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:30:28.665835 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/090bc9d2-7f8c-41b3-abd1-4b608891dba6-plugins-config-volume\") pod \"payload-processing-e2e-migrate-7b43a88f-6fdf74449d-99d59\" (UID: \"090bc9d2-7f8c-41b3-abd1-4b608891dba6\") " pod="openshift-ingress/payload-processing-e2e-migrate-7b43a88f-6fdf74449d-99d59" Sep 10 13:30:28.665964 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:30:28.665928 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1cce2cdc-ebd5-43f5-bb06-c0817da4e510-plugins-config-volume\") pod \"payload-pre-processing-e2e-migrate-7b43a88f-855895d994-7r7zc\" (UID: \"1cce2cdc-ebd5-43f5-bb06-c0817da4e510\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-7b43a88f-855895d994-7r7zc" Sep 10 13:30:28.666561 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:30:28.666541 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1cce2cdc-ebd5-43f5-bb06-c0817da4e510-plugins-config-volume\") pod \"payload-pre-processing-e2e-migrate-7b43a88f-855895d994-7r7zc\" (UID: \"1cce2cdc-ebd5-43f5-bb06-c0817da4e510\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-7b43a88f-855895d994-7r7zc" Sep 10 13:30:28.675298 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:30:28.675272 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5kbtc\" (UniqueName: \"kubernetes.io/projected/1cce2cdc-ebd5-43f5-bb06-c0817da4e510-kube-api-access-5kbtc\") pod \"payload-pre-processing-e2e-migrate-7b43a88f-855895d994-7r7zc\" (UID: \"1cce2cdc-ebd5-43f5-bb06-c0817da4e510\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-7b43a88f-855895d994-7r7zc" Sep 10 13:30:28.766940 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:30:28.766849 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zrpb2\" (UniqueName: \"kubernetes.io/projected/090bc9d2-7f8c-41b3-abd1-4b608891dba6-kube-api-access-zrpb2\") pod \"payload-processing-e2e-migrate-7b43a88f-6fdf74449d-99d59\" (UID: \"090bc9d2-7f8c-41b3-abd1-4b608891dba6\") " pod="openshift-ingress/payload-processing-e2e-migrate-7b43a88f-6fdf74449d-99d59" Sep 10 13:30:28.766940 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:30:28.766904 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/090bc9d2-7f8c-41b3-abd1-4b608891dba6-plugins-config-volume\") pod \"payload-processing-e2e-migrate-7b43a88f-6fdf74449d-99d59\" (UID: \"090bc9d2-7f8c-41b3-abd1-4b608891dba6\") " pod="openshift-ingress/payload-processing-e2e-migrate-7b43a88f-6fdf74449d-99d59" Sep 10 13:30:28.767585 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:30:28.767565 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/090bc9d2-7f8c-41b3-abd1-4b608891dba6-plugins-config-volume\") pod \"payload-processing-e2e-migrate-7b43a88f-6fdf74449d-99d59\" (UID: \"090bc9d2-7f8c-41b3-abd1-4b608891dba6\") " pod="openshift-ingress/payload-processing-e2e-migrate-7b43a88f-6fdf74449d-99d59" Sep 10 13:30:28.775739 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:30:28.775710 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zrpb2\" (UniqueName: \"kubernetes.io/projected/090bc9d2-7f8c-41b3-abd1-4b608891dba6-kube-api-access-zrpb2\") pod \"payload-processing-e2e-migrate-7b43a88f-6fdf74449d-99d59\" (UID: \"090bc9d2-7f8c-41b3-abd1-4b608891dba6\") " pod="openshift-ingress/payload-processing-e2e-migrate-7b43a88f-6fdf74449d-99d59" Sep 10 13:30:28.795720 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:30:28.795688 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-7b43a88f-855895d994-7r7zc" Sep 10 13:30:28.825958 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:30:28.825934 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-7b43a88f-6fdf74449d-99d59" Sep 10 13:30:29.083505 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:30:29.083478 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-7b43a88f-855895d994-7r7zc"] Sep 10 13:30:29.084414 ip-10-0-139-174 kubenswrapper[2557]: W0910 13:30:29.084387 2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1cce2cdc_ebd5_43f5_bb06_c0817da4e510.slice/crio-7916defb1aefec0857ac5d0a55db1905fa4cb531f45cc98c26116b0987c6962a WatchSource:0}: Error finding container 7916defb1aefec0857ac5d0a55db1905fa4cb531f45cc98c26116b0987c6962a: Status 404 returned error can't find the container with id 7916defb1aefec0857ac5d0a55db1905fa4cb531f45cc98c26116b0987c6962a Sep 10 13:30:29.137904 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:30:29.137870 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-7b43a88f-6fdf74449d-99d59"] Sep 10 13:30:29.139108 ip-10-0-139-174 kubenswrapper[2557]: W0910 13:30:29.139079 2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod090bc9d2_7f8c_41b3_abd1_4b608891dba6.slice/crio-e2f2944107d8a968eed3fae60e134275a471c8cedc85569da1e6d87a83ce4904 WatchSource:0}: Error finding container e2f2944107d8a968eed3fae60e134275a471c8cedc85569da1e6d87a83ce4904: Status 404 returned error can't find the container with id e2f2944107d8a968eed3fae60e134275a471c8cedc85569da1e6d87a83ce4904 Sep 10 13:30:29.471310 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:30:29.471274 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-7b43a88f-6fdf74449d-99d59" event={"ID":"090bc9d2-7f8c-41b3-abd1-4b608891dba6","Type":"ContainerStarted","Data":"f9a030eaa40b8a3bc643355f34b1c24893c6fac98ff44979096fda9663f18438"} Sep 10 13:30:29.471514 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:30:29.471338 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-7b43a88f-6fdf74449d-99d59" event={"ID":"090bc9d2-7f8c-41b3-abd1-4b608891dba6","Type":"ContainerStarted","Data":"e2f2944107d8a968eed3fae60e134275a471c8cedc85569da1e6d87a83ce4904"} Sep 10 13:30:29.471514 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:30:29.471377 2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-migrate-7b43a88f-6fdf74449d-99d59" Sep 10 13:30:29.473957 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:30:29.473926 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-7b43a88f-855895d994-7r7zc" event={"ID":"1cce2cdc-ebd5-43f5-bb06-c0817da4e510","Type":"ContainerStarted","Data":"bd57183ec30274b673ca3b5d2085e149605c23948d26c0515623c5be76fbed0b"} Sep 10 13:30:29.474105 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:30:29.473987 2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-migrate-7b43a88f-855895d994-7r7zc" Sep 10 13:30:29.474105 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:30:29.474013 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-7b43a88f-855895d994-7r7zc" event={"ID":"1cce2cdc-ebd5-43f5-bb06-c0817da4e510","Type":"ContainerStarted","Data":"7916defb1aefec0857ac5d0a55db1905fa4cb531f45cc98c26116b0987c6962a"} Sep 10 13:30:29.498198 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:30:29.498151 2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-migrate-7b43a88f-6fdf74449d-99d59" podStartSLOduration=1.4981375 podStartE2EDuration="1.4981375s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 13:30:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 13:30:29.494621517 +0000 UTC m=+789.801615930" watchObservedRunningTime="2026-09-10 13:30:29.4981375 +0000 UTC m=+789.805131913" Sep 10 13:30:29.518043 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:30:29.517995 2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-migrate-7b43a88f-855895d994-7r7zc" podStartSLOduration=1.5179802919999998 podStartE2EDuration="1.517980292s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 13:30:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 13:30:29.516387437 +0000 UTC m=+789.823381853" watchObservedRunningTime="2026-09-10 13:30:29.517980292 +0000 UTC m=+789.824974706" Sep 10 13:30:40.481469 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:30:40.481435 2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-migrate-7b43a88f-855895d994-7r7zc" Sep 10 13:30:40.481826 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:30:40.481630 2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-migrate-7b43a88f-6fdf74449d-99d59" Sep 10 13:30:52.573205 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:30:52.573102 2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-7b43a88f-l7dgj"] Sep 10 13:30:52.577098 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:30:52.577072 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-7b43a88f-l7dgj" Sep 10 13:30:52.581701 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:30:52.581680 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-wtblv\"" Sep 10 13:30:52.582158 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:30:52.582141 2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Sep 10 13:30:52.596194 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:30:52.596160 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-7b43a88f-l7dgj"] Sep 10 13:30:52.704524 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:30:52.704483 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3ff28ef5-897e-4807-b587-9ac9f2ca8a06-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-7b43a88f-l7dgj\" (UID: \"3ff28ef5-897e-4807-b587-9ac9f2ca8a06\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-7b43a88f-l7dgj" Sep 10 13:30:52.805868 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:30:52.805825 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3ff28ef5-897e-4807-b587-9ac9f2ca8a06-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-7b43a88f-l7dgj\" (UID: \"3ff28ef5-897e-4807-b587-9ac9f2ca8a06\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-7b43a88f-l7dgj" Sep 10 13:30:52.806787 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:30:52.806755 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3ff28ef5-897e-4807-b587-9ac9f2ca8a06-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-7b43a88f-l7dgj\" (UID: \"3ff28ef5-897e-4807-b587-9ac9f2ca8a06\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-7b43a88f-l7dgj" Sep 10 13:30:52.890964 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:30:52.890878 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-7b43a88f-l7dgj" Sep 10 13:30:53.185185 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:30:53.185158 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-7b43a88f-l7dgj"] Sep 10 13:30:53.187569 ip-10-0-139-174 kubenswrapper[2557]: W0910 13:30:53.187539 2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3ff28ef5_897e_4807_b587_9ac9f2ca8a06.slice/crio-b3a31f6c1df2784b6837e77b8bcbcdf25840cdb6ddd568a260e43496e96fef83 WatchSource:0}: Error finding container b3a31f6c1df2784b6837e77b8bcbcdf25840cdb6ddd568a260e43496e96fef83: Status 404 returned error can't find the container with id b3a31f6c1df2784b6837e77b8bcbcdf25840cdb6ddd568a260e43496e96fef83 Sep 10 13:30:53.632787 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:30:53.632749 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-7b43a88f-l7dgj" event={"ID":"3ff28ef5-897e-4807-b587-9ac9f2ca8a06","Type":"ContainerStarted","Data":"db4d4b6bc7082aed02445f8c95097451a32326293063d0e9ba40135d3b36ecde"} Sep 10 13:30:53.633181 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:30:53.632801 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-7b43a88f-l7dgj" event={"ID":"3ff28ef5-897e-4807-b587-9ac9f2ca8a06","Type":"ContainerStarted","Data":"b3a31f6c1df2784b6837e77b8bcbcdf25840cdb6ddd568a260e43496e96fef83"} Sep 10 13:30:53.663490 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:30:53.663445 2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-7b43a88f-l7dgj" podStartSLOduration=1.663411331 podStartE2EDuration="1.663411331s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 13:30:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 13:30:53.662537762 +0000 UTC m=+813.969532193" watchObservedRunningTime="2026-09-10 13:30:53.663411331 +0000 UTC m=+813.970405742" Sep 10 13:30:54.639445 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:30:54.639388 2557 generic.go:356] "Generic (PLEG): container finished" podID="3ff28ef5-897e-4807-b587-9ac9f2ca8a06" containerID="db4d4b6bc7082aed02445f8c95097451a32326293063d0e9ba40135d3b36ecde" exitCode=0 Sep 10 13:30:54.639863 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:30:54.639459 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-7b43a88f-l7dgj" event={"ID":"3ff28ef5-897e-4807-b587-9ac9f2ca8a06","Type":"ContainerDied","Data":"db4d4b6bc7082aed02445f8c95097451a32326293063d0e9ba40135d3b36ecde"} Sep 10 13:30:55.777399 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:30:55.777373 2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-7b43a88f-l7dgj" Sep 10 13:30:55.835813 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:30:55.835764 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3ff28ef5-897e-4807-b587-9ac9f2ca8a06-maas-api-service-ca\") pod \"3ff28ef5-897e-4807-b587-9ac9f2ca8a06\" (UID: \"3ff28ef5-897e-4807-b587-9ac9f2ca8a06\") " Sep 10 13:30:55.836123 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:30:55.836099 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3ff28ef5-897e-4807-b587-9ac9f2ca8a06-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "3ff28ef5-897e-4807-b587-9ac9f2ca8a06" (UID: "3ff28ef5-897e-4807-b587-9ac9f2ca8a06"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 10 13:30:55.937471 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:30:55.937381 2557 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3ff28ef5-897e-4807-b587-9ac9f2ca8a06-maas-api-service-ca\") on node \"ip-10-0-139-174.ec2.internal\" DevicePath \"\"" Sep 10 13:30:56.652592 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:30:56.652501 2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-7b43a88f-l7dgj" Sep 10 13:30:56.652592 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:30:56.652530 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-7b43a88f-l7dgj" event={"ID":"3ff28ef5-897e-4807-b587-9ac9f2ca8a06","Type":"ContainerDied","Data":"b3a31f6c1df2784b6837e77b8bcbcdf25840cdb6ddd568a260e43496e96fef83"} Sep 10 13:30:56.652592 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:30:56.652576 2557 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b3a31f6c1df2784b6837e77b8bcbcdf25840cdb6ddd568a260e43496e96fef83" Sep 10 13:30:59.691545 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:30:59.691513 2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-7b43a88f-6fdf74449d-99d59"] Sep 10 13:30:59.691957 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:30:59.691730 2557 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-migrate-7b43a88f-6fdf74449d-99d59" podUID="090bc9d2-7f8c-41b3-abd1-4b608891dba6" containerName="payload-processing" containerID="cri-o://f9a030eaa40b8a3bc643355f34b1c24893c6fac98ff44979096fda9663f18438" gracePeriod=30 Sep 10 13:30:59.712812 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:30:59.712777 2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-7b43a88f-855895d994-7r7zc"] Sep 10 13:30:59.713121 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:30:59.713066 2557 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-migrate-7b43a88f-855895d994-7r7zc" podUID="1cce2cdc-ebd5-43f5-bb06-c0817da4e510" containerName="payload-pre-processing" containerID="cri-o://bd57183ec30274b673ca3b5d2085e149605c23948d26c0515623c5be76fbed0b" gracePeriod=30 Sep 10 13:30:59.999592 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:30:59.999561 2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-7b43a88f-855895d994-7r7zc" Sep 10 13:31:00.002706 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:31:00.002683 2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-7b43a88f-6fdf74449d-99d59" Sep 10 13:31:00.075326 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:31:00.075294 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5kbtc\" (UniqueName: \"kubernetes.io/projected/1cce2cdc-ebd5-43f5-bb06-c0817da4e510-kube-api-access-5kbtc\") pod \"1cce2cdc-ebd5-43f5-bb06-c0817da4e510\" (UID: \"1cce2cdc-ebd5-43f5-bb06-c0817da4e510\") " Sep 10 13:31:00.075547 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:31:00.075393 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/090bc9d2-7f8c-41b3-abd1-4b608891dba6-plugins-config-volume\") pod \"090bc9d2-7f8c-41b3-abd1-4b608891dba6\" (UID: \"090bc9d2-7f8c-41b3-abd1-4b608891dba6\") " Sep 10 13:31:00.075621 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:31:00.075558 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1cce2cdc-ebd5-43f5-bb06-c0817da4e510-plugins-config-volume\") pod \"1cce2cdc-ebd5-43f5-bb06-c0817da4e510\" (UID: \"1cce2cdc-ebd5-43f5-bb06-c0817da4e510\") " Sep 10 13:31:00.075682 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:31:00.075670 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-zrpb2\" (UniqueName: \"kubernetes.io/projected/090bc9d2-7f8c-41b3-abd1-4b608891dba6-kube-api-access-zrpb2\") pod \"090bc9d2-7f8c-41b3-abd1-4b608891dba6\" (UID: \"090bc9d2-7f8c-41b3-abd1-4b608891dba6\") " Sep 10 13:31:00.075903 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:31:00.075875 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/090bc9d2-7f8c-41b3-abd1-4b608891dba6-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "090bc9d2-7f8c-41b3-abd1-4b608891dba6" (UID: "090bc9d2-7f8c-41b3-abd1-4b608891dba6"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 10 13:31:00.075976 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:31:00.075920 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1cce2cdc-ebd5-43f5-bb06-c0817da4e510-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "1cce2cdc-ebd5-43f5-bb06-c0817da4e510" (UID: "1cce2cdc-ebd5-43f5-bb06-c0817da4e510"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 10 13:31:00.076077 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:31:00.076061 2557 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/090bc9d2-7f8c-41b3-abd1-4b608891dba6-plugins-config-volume\") on node \"ip-10-0-139-174.ec2.internal\" DevicePath \"\"" Sep 10 13:31:00.076337 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:31:00.076086 2557 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1cce2cdc-ebd5-43f5-bb06-c0817da4e510-plugins-config-volume\") on node \"ip-10-0-139-174.ec2.internal\" DevicePath \"\"" Sep 10 13:31:00.077904 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:31:00.077880 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1cce2cdc-ebd5-43f5-bb06-c0817da4e510-kube-api-access-5kbtc" (OuterVolumeSpecName: "kube-api-access-5kbtc") pod "1cce2cdc-ebd5-43f5-bb06-c0817da4e510" (UID: "1cce2cdc-ebd5-43f5-bb06-c0817da4e510"). InnerVolumeSpecName "kube-api-access-5kbtc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 10 13:31:00.078024 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:31:00.077903 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/090bc9d2-7f8c-41b3-abd1-4b608891dba6-kube-api-access-zrpb2" (OuterVolumeSpecName: "kube-api-access-zrpb2") pod "090bc9d2-7f8c-41b3-abd1-4b608891dba6" (UID: "090bc9d2-7f8c-41b3-abd1-4b608891dba6"). InnerVolumeSpecName "kube-api-access-zrpb2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 10 13:31:00.177487 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:31:00.177408 2557 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zrpb2\" (UniqueName: \"kubernetes.io/projected/090bc9d2-7f8c-41b3-abd1-4b608891dba6-kube-api-access-zrpb2\") on node \"ip-10-0-139-174.ec2.internal\" DevicePath \"\"" Sep 10 13:31:00.177487 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:31:00.177481 2557 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5kbtc\" (UniqueName: \"kubernetes.io/projected/1cce2cdc-ebd5-43f5-bb06-c0817da4e510-kube-api-access-5kbtc\") on node \"ip-10-0-139-174.ec2.internal\" DevicePath \"\"" Sep 10 13:31:00.511209 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:31:00.511105 2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-7b43a88f-l7dgj"] Sep 10 13:31:00.525975 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:31:00.525928 2557 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-7b43a88f-l7dgj"] Sep 10 13:31:00.678662 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:31:00.678625 2557 generic.go:356] "Generic (PLEG): container finished" podID="090bc9d2-7f8c-41b3-abd1-4b608891dba6" containerID="f9a030eaa40b8a3bc643355f34b1c24893c6fac98ff44979096fda9663f18438" exitCode=0 Sep 10 13:31:00.678858 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:31:00.678689 2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-7b43a88f-6fdf74449d-99d59" Sep 10 13:31:00.678858 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:31:00.678717 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-7b43a88f-6fdf74449d-99d59" event={"ID":"090bc9d2-7f8c-41b3-abd1-4b608891dba6","Type":"ContainerDied","Data":"f9a030eaa40b8a3bc643355f34b1c24893c6fac98ff44979096fda9663f18438"} Sep 10 13:31:00.678858 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:31:00.678809 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-7b43a88f-6fdf74449d-99d59" event={"ID":"090bc9d2-7f8c-41b3-abd1-4b608891dba6","Type":"ContainerDied","Data":"e2f2944107d8a968eed3fae60e134275a471c8cedc85569da1e6d87a83ce4904"} Sep 10 13:31:00.678858 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:31:00.678841 2557 scope.go:117] "RemoveContainer" containerID="f9a030eaa40b8a3bc643355f34b1c24893c6fac98ff44979096fda9663f18438" Sep 10 13:31:00.680390 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:31:00.680365 2557 generic.go:356] "Generic (PLEG): container finished" podID="1cce2cdc-ebd5-43f5-bb06-c0817da4e510" containerID="bd57183ec30274b673ca3b5d2085e149605c23948d26c0515623c5be76fbed0b" exitCode=0 Sep 10 13:31:00.680506 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:31:00.680458 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-7b43a88f-855895d994-7r7zc" event={"ID":"1cce2cdc-ebd5-43f5-bb06-c0817da4e510","Type":"ContainerDied","Data":"bd57183ec30274b673ca3b5d2085e149605c23948d26c0515623c5be76fbed0b"} Sep 10 13:31:00.680506 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:31:00.680501 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-7b43a88f-855895d994-7r7zc" event={"ID":"1cce2cdc-ebd5-43f5-bb06-c0817da4e510","Type":"ContainerDied","Data":"7916defb1aefec0857ac5d0a55db1905fa4cb531f45cc98c26116b0987c6962a"} Sep 10 13:31:00.680638 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:31:00.680620 2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-7b43a88f-855895d994-7r7zc" Sep 10 13:31:00.688474 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:31:00.688449 2557 scope.go:117] "RemoveContainer" containerID="f9a030eaa40b8a3bc643355f34b1c24893c6fac98ff44979096fda9663f18438" Sep 10 13:31:00.688790 ip-10-0-139-174 kubenswrapper[2557]: E0910 13:31:00.688768 2557 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f9a030eaa40b8a3bc643355f34b1c24893c6fac98ff44979096fda9663f18438\": container with ID starting with f9a030eaa40b8a3bc643355f34b1c24893c6fac98ff44979096fda9663f18438 not found: ID does not exist" containerID="f9a030eaa40b8a3bc643355f34b1c24893c6fac98ff44979096fda9663f18438" Sep 10 13:31:00.688856 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:31:00.688797 2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f9a030eaa40b8a3bc643355f34b1c24893c6fac98ff44979096fda9663f18438"} err="failed to get container status \"f9a030eaa40b8a3bc643355f34b1c24893c6fac98ff44979096fda9663f18438\": rpc error: code = NotFound desc = could not find container \"f9a030eaa40b8a3bc643355f34b1c24893c6fac98ff44979096fda9663f18438\": container with ID starting with f9a030eaa40b8a3bc643355f34b1c24893c6fac98ff44979096fda9663f18438 not found: ID does not exist" Sep 10 13:31:00.688856 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:31:00.688815 2557 scope.go:117] "RemoveContainer" containerID="bd57183ec30274b673ca3b5d2085e149605c23948d26c0515623c5be76fbed0b" Sep 10 13:31:00.698041 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:31:00.697882 2557 scope.go:117] "RemoveContainer" containerID="bd57183ec30274b673ca3b5d2085e149605c23948d26c0515623c5be76fbed0b" Sep 10 13:31:00.698261 ip-10-0-139-174 kubenswrapper[2557]: E0910 13:31:00.698173 2557 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bd57183ec30274b673ca3b5d2085e149605c23948d26c0515623c5be76fbed0b\": container with ID starting with bd57183ec30274b673ca3b5d2085e149605c23948d26c0515623c5be76fbed0b not found: ID does not exist" containerID="bd57183ec30274b673ca3b5d2085e149605c23948d26c0515623c5be76fbed0b" Sep 10 13:31:00.698261 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:31:00.698196 2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bd57183ec30274b673ca3b5d2085e149605c23948d26c0515623c5be76fbed0b"} err="failed to get container status \"bd57183ec30274b673ca3b5d2085e149605c23948d26c0515623c5be76fbed0b\": rpc error: code = NotFound desc = could not find container \"bd57183ec30274b673ca3b5d2085e149605c23948d26c0515623c5be76fbed0b\": container with ID starting with bd57183ec30274b673ca3b5d2085e149605c23948d26c0515623c5be76fbed0b not found: ID does not exist" Sep 10 13:31:00.731585 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:31:00.731553 2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-7b43a88f-6fdf74449d-99d59"] Sep 10 13:31:00.738699 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:31:00.738666 2557 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-7b43a88f-6fdf74449d-99d59"] Sep 10 13:31:00.759383 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:31:00.759352 2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-7b43a88f-855895d994-7r7zc"] Sep 10 13:31:00.767819 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:31:00.767726 2557 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-7b43a88f-855895d994-7r7zc"] Sep 10 13:31:01.704220 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:31:01.704181 2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ait-300680f3-openshift-default-584b5ff64d-t2ccg"] Sep 10 13:31:01.704818 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:31:01.704798 2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="090bc9d2-7f8c-41b3-abd1-4b608891dba6" containerName="payload-processing" Sep 10 13:31:01.704924 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:31:01.704820 2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="090bc9d2-7f8c-41b3-abd1-4b608891dba6" containerName="payload-processing" Sep 10 13:31:01.704924 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:31:01.704840 2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1cce2cdc-ebd5-43f5-bb06-c0817da4e510" containerName="payload-pre-processing" Sep 10 13:31:01.704924 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:31:01.704848 2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="1cce2cdc-ebd5-43f5-bb06-c0817da4e510" containerName="payload-pre-processing" Sep 10 13:31:01.704924 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:31:01.704876 2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3ff28ef5-897e-4807-b587-9ac9f2ca8a06" containerName="revoke-keys" Sep 10 13:31:01.704924 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:31:01.704884 2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="3ff28ef5-897e-4807-b587-9ac9f2ca8a06" containerName="revoke-keys" Sep 10 13:31:01.705162 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:31:01.704984 2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="3ff28ef5-897e-4807-b587-9ac9f2ca8a06" containerName="revoke-keys" Sep 10 13:31:01.705162 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:31:01.704997 2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="090bc9d2-7f8c-41b3-abd1-4b608891dba6" containerName="payload-processing" Sep 10 13:31:01.705162 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:31:01.705007 2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="1cce2cdc-ebd5-43f5-bb06-c0817da4e510" containerName="payload-pre-processing" Sep 10 13:31:01.709818 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:31:01.709797 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-300680f3-openshift-default-584b5ff64d-t2ccg" Sep 10 13:31:01.717348 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:31:01.717329 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ait-300680f3-openshift-default-dockercfg-kfl4h\"" Sep 10 13:31:01.735490 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:31:01.735462 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-300680f3-openshift-default-584b5ff64d-t2ccg"] Sep 10 13:31:01.795925 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:31:01.795888 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/27bdc7ae-a8d0-43fb-8d70-7a247ef9db39-workload-certs\") pod \"e2e-ait-300680f3-openshift-default-584b5ff64d-t2ccg\" (UID: \"27bdc7ae-a8d0-43fb-8d70-7a247ef9db39\") " pod="openshift-ingress/e2e-ait-300680f3-openshift-default-584b5ff64d-t2ccg" Sep 10 13:31:01.796102 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:31:01.795931 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/27bdc7ae-a8d0-43fb-8d70-7a247ef9db39-istio-data\") pod \"e2e-ait-300680f3-openshift-default-584b5ff64d-t2ccg\" (UID: \"27bdc7ae-a8d0-43fb-8d70-7a247ef9db39\") " pod="openshift-ingress/e2e-ait-300680f3-openshift-default-584b5ff64d-t2ccg" Sep 10 13:31:01.796102 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:31:01.795956 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/27bdc7ae-a8d0-43fb-8d70-7a247ef9db39-istio-token\") pod \"e2e-ait-300680f3-openshift-default-584b5ff64d-t2ccg\" (UID: \"27bdc7ae-a8d0-43fb-8d70-7a247ef9db39\") " pod="openshift-ingress/e2e-ait-300680f3-openshift-default-584b5ff64d-t2ccg" Sep 10 13:31:01.796102 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:31:01.796007 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/27bdc7ae-a8d0-43fb-8d70-7a247ef9db39-istio-envoy\") pod \"e2e-ait-300680f3-openshift-default-584b5ff64d-t2ccg\" (UID: \"27bdc7ae-a8d0-43fb-8d70-7a247ef9db39\") " pod="openshift-ingress/e2e-ait-300680f3-openshift-default-584b5ff64d-t2ccg" Sep 10 13:31:01.796102 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:31:01.796044 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/27bdc7ae-a8d0-43fb-8d70-7a247ef9db39-credential-socket\") pod \"e2e-ait-300680f3-openshift-default-584b5ff64d-t2ccg\" (UID: \"27bdc7ae-a8d0-43fb-8d70-7a247ef9db39\") " pod="openshift-ingress/e2e-ait-300680f3-openshift-default-584b5ff64d-t2ccg" Sep 10 13:31:01.796102 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:31:01.796101 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/27bdc7ae-a8d0-43fb-8d70-7a247ef9db39-workload-socket\") pod \"e2e-ait-300680f3-openshift-default-584b5ff64d-t2ccg\" (UID: \"27bdc7ae-a8d0-43fb-8d70-7a247ef9db39\") " pod="openshift-ingress/e2e-ait-300680f3-openshift-default-584b5ff64d-t2ccg" Sep 10 13:31:01.796394 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:31:01.796226 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/27bdc7ae-a8d0-43fb-8d70-7a247ef9db39-istiod-ca-cert\") pod \"e2e-ait-300680f3-openshift-default-584b5ff64d-t2ccg\" (UID: \"27bdc7ae-a8d0-43fb-8d70-7a247ef9db39\") " pod="openshift-ingress/e2e-ait-300680f3-openshift-default-584b5ff64d-t2ccg" Sep 10 13:31:01.796394 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:31:01.796266 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/27bdc7ae-a8d0-43fb-8d70-7a247ef9db39-istio-podinfo\") pod \"e2e-ait-300680f3-openshift-default-584b5ff64d-t2ccg\" (UID: \"27bdc7ae-a8d0-43fb-8d70-7a247ef9db39\") " pod="openshift-ingress/e2e-ait-300680f3-openshift-default-584b5ff64d-t2ccg" Sep 10 13:31:01.796394 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:31:01.796352 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cqfv4\" (UniqueName: \"kubernetes.io/projected/27bdc7ae-a8d0-43fb-8d70-7a247ef9db39-kube-api-access-cqfv4\") pod \"e2e-ait-300680f3-openshift-default-584b5ff64d-t2ccg\" (UID: \"27bdc7ae-a8d0-43fb-8d70-7a247ef9db39\") " pod="openshift-ingress/e2e-ait-300680f3-openshift-default-584b5ff64d-t2ccg" Sep 10 13:31:01.897554 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:31:01.897500 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cqfv4\" (UniqueName: \"kubernetes.io/projected/27bdc7ae-a8d0-43fb-8d70-7a247ef9db39-kube-api-access-cqfv4\") pod \"e2e-ait-300680f3-openshift-default-584b5ff64d-t2ccg\" (UID: \"27bdc7ae-a8d0-43fb-8d70-7a247ef9db39\") " pod="openshift-ingress/e2e-ait-300680f3-openshift-default-584b5ff64d-t2ccg" Sep 10 13:31:01.897754 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:31:01.897563 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/27bdc7ae-a8d0-43fb-8d70-7a247ef9db39-workload-certs\") pod \"e2e-ait-300680f3-openshift-default-584b5ff64d-t2ccg\" (UID: \"27bdc7ae-a8d0-43fb-8d70-7a247ef9db39\") " pod="openshift-ingress/e2e-ait-300680f3-openshift-default-584b5ff64d-t2ccg" Sep 10 13:31:01.897754 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:31:01.897592 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/27bdc7ae-a8d0-43fb-8d70-7a247ef9db39-istio-data\") pod \"e2e-ait-300680f3-openshift-default-584b5ff64d-t2ccg\" (UID: \"27bdc7ae-a8d0-43fb-8d70-7a247ef9db39\") " pod="openshift-ingress/e2e-ait-300680f3-openshift-default-584b5ff64d-t2ccg" Sep 10 13:31:01.897754 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:31:01.897618 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/27bdc7ae-a8d0-43fb-8d70-7a247ef9db39-istio-token\") pod \"e2e-ait-300680f3-openshift-default-584b5ff64d-t2ccg\" (UID: \"27bdc7ae-a8d0-43fb-8d70-7a247ef9db39\") " pod="openshift-ingress/e2e-ait-300680f3-openshift-default-584b5ff64d-t2ccg" Sep 10 13:31:01.897754 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:31:01.897649 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/27bdc7ae-a8d0-43fb-8d70-7a247ef9db39-istio-envoy\") pod \"e2e-ait-300680f3-openshift-default-584b5ff64d-t2ccg\" (UID: \"27bdc7ae-a8d0-43fb-8d70-7a247ef9db39\") " pod="openshift-ingress/e2e-ait-300680f3-openshift-default-584b5ff64d-t2ccg" Sep 10 13:31:01.897754 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:31:01.897680 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/27bdc7ae-a8d0-43fb-8d70-7a247ef9db39-credential-socket\") pod \"e2e-ait-300680f3-openshift-default-584b5ff64d-t2ccg\" (UID: \"27bdc7ae-a8d0-43fb-8d70-7a247ef9db39\") " pod="openshift-ingress/e2e-ait-300680f3-openshift-default-584b5ff64d-t2ccg" Sep 10 13:31:01.897754 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:31:01.897755 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/27bdc7ae-a8d0-43fb-8d70-7a247ef9db39-workload-socket\") pod \"e2e-ait-300680f3-openshift-default-584b5ff64d-t2ccg\" (UID: \"27bdc7ae-a8d0-43fb-8d70-7a247ef9db39\") " pod="openshift-ingress/e2e-ait-300680f3-openshift-default-584b5ff64d-t2ccg" Sep 10 13:31:01.898065 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:31:01.897867 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/27bdc7ae-a8d0-43fb-8d70-7a247ef9db39-istiod-ca-cert\") pod \"e2e-ait-300680f3-openshift-default-584b5ff64d-t2ccg\" (UID: \"27bdc7ae-a8d0-43fb-8d70-7a247ef9db39\") " pod="openshift-ingress/e2e-ait-300680f3-openshift-default-584b5ff64d-t2ccg" Sep 10 13:31:01.898065 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:31:01.897904 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/27bdc7ae-a8d0-43fb-8d70-7a247ef9db39-istio-podinfo\") pod \"e2e-ait-300680f3-openshift-default-584b5ff64d-t2ccg\" (UID: \"27bdc7ae-a8d0-43fb-8d70-7a247ef9db39\") " pod="openshift-ingress/e2e-ait-300680f3-openshift-default-584b5ff64d-t2ccg" Sep 10 13:31:01.898653 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:31:01.898624 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/27bdc7ae-a8d0-43fb-8d70-7a247ef9db39-istio-data\") pod \"e2e-ait-300680f3-openshift-default-584b5ff64d-t2ccg\" (UID: \"27bdc7ae-a8d0-43fb-8d70-7a247ef9db39\") " pod="openshift-ingress/e2e-ait-300680f3-openshift-default-584b5ff64d-t2ccg" Sep 10 13:31:01.900622 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:31:01.899224 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/27bdc7ae-a8d0-43fb-8d70-7a247ef9db39-workload-socket\") pod \"e2e-ait-300680f3-openshift-default-584b5ff64d-t2ccg\" (UID: \"27bdc7ae-a8d0-43fb-8d70-7a247ef9db39\") " pod="openshift-ingress/e2e-ait-300680f3-openshift-default-584b5ff64d-t2ccg" Sep 10 13:31:01.900873 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:31:01.899479 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/27bdc7ae-a8d0-43fb-8d70-7a247ef9db39-credential-socket\") pod \"e2e-ait-300680f3-openshift-default-584b5ff64d-t2ccg\" (UID: \"27bdc7ae-a8d0-43fb-8d70-7a247ef9db39\") " pod="openshift-ingress/e2e-ait-300680f3-openshift-default-584b5ff64d-t2ccg" Sep 10 13:31:01.900962 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:31:01.900090 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/27bdc7ae-a8d0-43fb-8d70-7a247ef9db39-istiod-ca-cert\") pod \"e2e-ait-300680f3-openshift-default-584b5ff64d-t2ccg\" (UID: \"27bdc7ae-a8d0-43fb-8d70-7a247ef9db39\") " pod="openshift-ingress/e2e-ait-300680f3-openshift-default-584b5ff64d-t2ccg" Sep 10 13:31:01.900962 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:31:01.900363 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/27bdc7ae-a8d0-43fb-8d70-7a247ef9db39-workload-certs\") pod \"e2e-ait-300680f3-openshift-default-584b5ff64d-t2ccg\" (UID: \"27bdc7ae-a8d0-43fb-8d70-7a247ef9db39\") " pod="openshift-ingress/e2e-ait-300680f3-openshift-default-584b5ff64d-t2ccg" Sep 10 13:31:01.901130 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:31:01.901108 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/27bdc7ae-a8d0-43fb-8d70-7a247ef9db39-istio-envoy\") pod \"e2e-ait-300680f3-openshift-default-584b5ff64d-t2ccg\" (UID: \"27bdc7ae-a8d0-43fb-8d70-7a247ef9db39\") " pod="openshift-ingress/e2e-ait-300680f3-openshift-default-584b5ff64d-t2ccg" Sep 10 13:31:01.902362 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:31:01.902189 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/27bdc7ae-a8d0-43fb-8d70-7a247ef9db39-istio-podinfo\") pod \"e2e-ait-300680f3-openshift-default-584b5ff64d-t2ccg\" (UID: \"27bdc7ae-a8d0-43fb-8d70-7a247ef9db39\") " pod="openshift-ingress/e2e-ait-300680f3-openshift-default-584b5ff64d-t2ccg" Sep 10 13:31:01.920872 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:31:01.920093 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/27bdc7ae-a8d0-43fb-8d70-7a247ef9db39-istio-token\") pod \"e2e-ait-300680f3-openshift-default-584b5ff64d-t2ccg\" (UID: \"27bdc7ae-a8d0-43fb-8d70-7a247ef9db39\") " pod="openshift-ingress/e2e-ait-300680f3-openshift-default-584b5ff64d-t2ccg" Sep 10 13:31:01.920872 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:31:01.920798 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cqfv4\" (UniqueName: \"kubernetes.io/projected/27bdc7ae-a8d0-43fb-8d70-7a247ef9db39-kube-api-access-cqfv4\") pod \"e2e-ait-300680f3-openshift-default-584b5ff64d-t2ccg\" (UID: \"27bdc7ae-a8d0-43fb-8d70-7a247ef9db39\") " pod="openshift-ingress/e2e-ait-300680f3-openshift-default-584b5ff64d-t2ccg" Sep 10 13:31:02.020352 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:31:02.020248 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-300680f3-openshift-default-584b5ff64d-t2ccg" Sep 10 13:31:02.222177 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:31:02.222140 2557 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="090bc9d2-7f8c-41b3-abd1-4b608891dba6" path="/var/lib/kubelet/pods/090bc9d2-7f8c-41b3-abd1-4b608891dba6/volumes" Sep 10 13:31:02.222566 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:31:02.222548 2557 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1cce2cdc-ebd5-43f5-bb06-c0817da4e510" path="/var/lib/kubelet/pods/1cce2cdc-ebd5-43f5-bb06-c0817da4e510/volumes" Sep 10 13:31:02.222851 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:31:02.222836 2557 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3ff28ef5-897e-4807-b587-9ac9f2ca8a06" path="/var/lib/kubelet/pods/3ff28ef5-897e-4807-b587-9ac9f2ca8a06/volumes" Sep 10 13:31:02.325567 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:31:02.325537 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-300680f3-openshift-default-584b5ff64d-t2ccg"] Sep 10 13:31:02.328245 ip-10-0-139-174 kubenswrapper[2557]: W0910 13:31:02.328212 2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod27bdc7ae_a8d0_43fb_8d70_7a247ef9db39.slice/crio-c9651c917370cb19a84ba48a4656d670d1871375fdca238799b0344d8a1a9d2a WatchSource:0}: Error finding container c9651c917370cb19a84ba48a4656d670d1871375fdca238799b0344d8a1a9d2a: Status 404 returned error can't find the container with id c9651c917370cb19a84ba48a4656d670d1871375fdca238799b0344d8a1a9d2a Sep 10 13:31:02.330913 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:31:02.330870 2557 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892152Ki","pods":"250"} Sep 10 13:31:02.331033 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:31:02.330973 2557 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892152Ki","pods":"250"} Sep 10 13:31:02.331033 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:31:02.331019 2557 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892152Ki","pods":"250"} Sep 10 13:31:02.702958 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:31:02.702922 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-300680f3-openshift-default-584b5ff64d-t2ccg" event={"ID":"27bdc7ae-a8d0-43fb-8d70-7a247ef9db39","Type":"ContainerStarted","Data":"95f12407cae6137473e8d381f747c194eb74293e0bb2e45deabefdc156c7f394"} Sep 10 13:31:02.703184 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:31:02.702994 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-300680f3-openshift-default-584b5ff64d-t2ccg" event={"ID":"27bdc7ae-a8d0-43fb-8d70-7a247ef9db39","Type":"ContainerStarted","Data":"c9651c917370cb19a84ba48a4656d670d1871375fdca238799b0344d8a1a9d2a"} Sep 10 13:31:02.749243 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:31:02.749173 2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-ait-300680f3-openshift-default-584b5ff64d-t2ccg" podStartSLOduration=1.749148921 podStartE2EDuration="1.749148921s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 13:31:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 13:31:02.743672748 +0000 UTC m=+823.050667163" watchObservedRunningTime="2026-09-10 13:31:02.749148921 +0000 UTC m=+823.056143341" Sep 10 13:31:03.020950 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:31:03.020860 2557 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-300680f3-openshift-default-584b5ff64d-t2ccg" Sep 10 13:31:03.026184 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:31:03.026162 2557 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-300680f3-openshift-default-584b5ff64d-t2ccg" Sep 10 13:31:03.708573 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:31:03.708543 2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-300680f3-openshift-default-584b5ff64d-t2ccg" Sep 10 13:31:03.709619 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:31:03.709602 2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-300680f3-openshift-default-584b5ff64d-t2ccg" Sep 10 13:31:07.108982 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:31:07.108945 2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-300680f3-547d4f8c46-zflds"] Sep 10 13:31:07.112656 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:31:07.112637 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-300680f3-547d4f8c46-zflds" Sep 10 13:31:07.115338 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:31:07.115319 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ait-300680f3-dockercfg-zv7ng\"" Sep 10 13:31:07.115756 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:31:07.115740 2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ait-300680f3\"" Sep 10 13:31:07.139257 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:31:07.139227 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-300680f3-547d4f8c46-zflds"] Sep 10 13:31:07.253891 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:31:07.253853 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m65gt\" (UniqueName: \"kubernetes.io/projected/224fe991-782f-4f11-a321-3e6c4f03c0b8-kube-api-access-m65gt\") pod \"payload-pre-processing-e2e-ait-300680f3-547d4f8c46-zflds\" (UID: \"224fe991-782f-4f11-a321-3e6c4f03c0b8\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-300680f3-547d4f8c46-zflds" Sep 10 13:31:07.254068 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:31:07.253943 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/224fe991-782f-4f11-a321-3e6c4f03c0b8-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-300680f3-547d4f8c46-zflds\" (UID: \"224fe991-782f-4f11-a321-3e6c4f03c0b8\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-300680f3-547d4f8c46-zflds" Sep 10 13:31:07.355257 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:31:07.355222 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m65gt\" (UniqueName: \"kubernetes.io/projected/224fe991-782f-4f11-a321-3e6c4f03c0b8-kube-api-access-m65gt\") pod \"payload-pre-processing-e2e-ait-300680f3-547d4f8c46-zflds\" (UID: \"224fe991-782f-4f11-a321-3e6c4f03c0b8\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-300680f3-547d4f8c46-zflds" Sep 10 13:31:07.355488 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:31:07.355366 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/224fe991-782f-4f11-a321-3e6c4f03c0b8-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-300680f3-547d4f8c46-zflds\" (UID: \"224fe991-782f-4f11-a321-3e6c4f03c0b8\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-300680f3-547d4f8c46-zflds" Sep 10 13:31:07.356102 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:31:07.356079 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/224fe991-782f-4f11-a321-3e6c4f03c0b8-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-300680f3-547d4f8c46-zflds\" (UID: \"224fe991-782f-4f11-a321-3e6c4f03c0b8\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-300680f3-547d4f8c46-zflds" Sep 10 13:31:07.369467 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:31:07.369383 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-m65gt\" (UniqueName: \"kubernetes.io/projected/224fe991-782f-4f11-a321-3e6c4f03c0b8-kube-api-access-m65gt\") pod \"payload-pre-processing-e2e-ait-300680f3-547d4f8c46-zflds\" (UID: \"224fe991-782f-4f11-a321-3e6c4f03c0b8\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-300680f3-547d4f8c46-zflds" Sep 10 13:31:07.423939 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:31:07.423903 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-300680f3-547d4f8c46-zflds" Sep 10 13:31:07.721282 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:31:07.721250 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-300680f3-547d4f8c46-zflds"] Sep 10 13:31:07.724848 ip-10-0-139-174 kubenswrapper[2557]: W0910 13:31:07.724821 2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod224fe991_782f_4f11_a321_3e6c4f03c0b8.slice/crio-052643ad843956db0d5ac7351dd54183e0e68e82442ed6f5017d962e425430bf WatchSource:0}: Error finding container 052643ad843956db0d5ac7351dd54183e0e68e82442ed6f5017d962e425430bf: Status 404 returned error can't find the container with id 052643ad843956db0d5ac7351dd54183e0e68e82442ed6f5017d962e425430bf Sep 10 13:31:07.732563 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:31:07.732539 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-300680f3-547d4f8c46-zflds" event={"ID":"224fe991-782f-4f11-a321-3e6c4f03c0b8","Type":"ContainerStarted","Data":"052643ad843956db0d5ac7351dd54183e0e68e82442ed6f5017d962e425430bf"} Sep 10 13:31:08.739899 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:31:08.739861 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-300680f3-547d4f8c46-zflds" event={"ID":"224fe991-782f-4f11-a321-3e6c4f03c0b8","Type":"ContainerStarted","Data":"982f36cce16297922720f34c5be079b861fde286f0e58cd6509a67b8e5890c32"} Sep 10 13:31:08.740333 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:31:08.739999 2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-300680f3-547d4f8c46-zflds" Sep 10 13:31:08.772487 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:31:08.772443 2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ait-300680f3-547d4f8c46-zflds" podStartSLOduration=1.7724082700000001 podStartE2EDuration="1.77240827s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 13:31:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 13:31:08.770725178 +0000 UTC m=+829.077719593" watchObservedRunningTime="2026-09-10 13:31:08.77240827 +0000 UTC m=+829.079402684" Sep 10 13:31:18.220307 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:31:18.220279 2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-a-39df6e/test-model-39df6e-kserve-698d88cd99-xgkz7"] Sep 10 13:31:18.223763 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:31:18.223748 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-39df6e/test-model-39df6e-kserve-698d88cd99-xgkz7" Sep 10 13:31:18.228184 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:31:18.228161 2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-39df6e\"/\"kube-root-ca.crt\"" Sep 10 13:31:18.228616 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:31:18.228597 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-39df6e\"/\"test-model-39df6e-kserve-self-signed-certs\"" Sep 10 13:31:18.228729 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:31:18.228709 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-39df6e\"/\"default-dockercfg-t8sc8\"" Sep 10 13:31:18.228872 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:31:18.228748 2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-39df6e\"/\"openshift-service-ca.crt\"" Sep 10 13:31:18.236876 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:31:18.236854 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-39df6e/test-model-39df6e-kserve-698d88cd99-xgkz7"] Sep 10 13:31:18.360653 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:31:18.360620 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/76f7bb42-a548-41cf-9bea-0f2734d74b24-dshm\") pod \"test-model-39df6e-kserve-698d88cd99-xgkz7\" (UID: \"76f7bb42-a548-41cf-9bea-0f2734d74b24\") " pod="ai-tenant-e2e-inf-a-39df6e/test-model-39df6e-kserve-698d88cd99-xgkz7" Sep 10 13:31:18.360801 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:31:18.360663 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/76f7bb42-a548-41cf-9bea-0f2734d74b24-model-cache\") pod \"test-model-39df6e-kserve-698d88cd99-xgkz7\" (UID: \"76f7bb42-a548-41cf-9bea-0f2734d74b24\") " pod="ai-tenant-e2e-inf-a-39df6e/test-model-39df6e-kserve-698d88cd99-xgkz7" Sep 10 13:31:18.360801 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:31:18.360700 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/76f7bb42-a548-41cf-9bea-0f2734d74b24-home\") pod \"test-model-39df6e-kserve-698d88cd99-xgkz7\" (UID: \"76f7bb42-a548-41cf-9bea-0f2734d74b24\") " pod="ai-tenant-e2e-inf-a-39df6e/test-model-39df6e-kserve-698d88cd99-xgkz7" Sep 10 13:31:18.360801 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:31:18.360760 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/76f7bb42-a548-41cf-9bea-0f2734d74b24-tls-certs\") pod \"test-model-39df6e-kserve-698d88cd99-xgkz7\" (UID: \"76f7bb42-a548-41cf-9bea-0f2734d74b24\") " pod="ai-tenant-e2e-inf-a-39df6e/test-model-39df6e-kserve-698d88cd99-xgkz7" Sep 10 13:31:18.360801 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:31:18.360791 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/76f7bb42-a548-41cf-9bea-0f2734d74b24-tmp-dir\") pod \"test-model-39df6e-kserve-698d88cd99-xgkz7\" (UID: \"76f7bb42-a548-41cf-9bea-0f2734d74b24\") " pod="ai-tenant-e2e-inf-a-39df6e/test-model-39df6e-kserve-698d88cd99-xgkz7" Sep 10 13:31:18.360942 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:31:18.360810 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xg628\" (UniqueName: \"kubernetes.io/projected/76f7bb42-a548-41cf-9bea-0f2734d74b24-kube-api-access-xg628\") pod \"test-model-39df6e-kserve-698d88cd99-xgkz7\" (UID: \"76f7bb42-a548-41cf-9bea-0f2734d74b24\") " pod="ai-tenant-e2e-inf-a-39df6e/test-model-39df6e-kserve-698d88cd99-xgkz7" Sep 10 13:31:18.461586 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:31:18.461550 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/76f7bb42-a548-41cf-9bea-0f2734d74b24-dshm\") pod \"test-model-39df6e-kserve-698d88cd99-xgkz7\" (UID: \"76f7bb42-a548-41cf-9bea-0f2734d74b24\") " pod="ai-tenant-e2e-inf-a-39df6e/test-model-39df6e-kserve-698d88cd99-xgkz7" Sep 10 13:31:18.461871 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:31:18.461848 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/76f7bb42-a548-41cf-9bea-0f2734d74b24-model-cache\") pod \"test-model-39df6e-kserve-698d88cd99-xgkz7\" (UID: \"76f7bb42-a548-41cf-9bea-0f2734d74b24\") " pod="ai-tenant-e2e-inf-a-39df6e/test-model-39df6e-kserve-698d88cd99-xgkz7" Sep 10 13:31:18.462030 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:31:18.462015 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/76f7bb42-a548-41cf-9bea-0f2734d74b24-home\") pod \"test-model-39df6e-kserve-698d88cd99-xgkz7\" (UID: \"76f7bb42-a548-41cf-9bea-0f2734d74b24\") " pod="ai-tenant-e2e-inf-a-39df6e/test-model-39df6e-kserve-698d88cd99-xgkz7" Sep 10 13:31:18.462233 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:31:18.462218 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/76f7bb42-a548-41cf-9bea-0f2734d74b24-tls-certs\") pod \"test-model-39df6e-kserve-698d88cd99-xgkz7\" (UID: \"76f7bb42-a548-41cf-9bea-0f2734d74b24\") " pod="ai-tenant-e2e-inf-a-39df6e/test-model-39df6e-kserve-698d88cd99-xgkz7" Sep 10 13:31:18.462867 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:31:18.462340 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/76f7bb42-a548-41cf-9bea-0f2734d74b24-model-cache\") pod \"test-model-39df6e-kserve-698d88cd99-xgkz7\" (UID: \"76f7bb42-a548-41cf-9bea-0f2734d74b24\") " pod="ai-tenant-e2e-inf-a-39df6e/test-model-39df6e-kserve-698d88cd99-xgkz7" Sep 10 13:31:18.462867 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:31:18.462476 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/76f7bb42-a548-41cf-9bea-0f2734d74b24-home\") pod \"test-model-39df6e-kserve-698d88cd99-xgkz7\" (UID: \"76f7bb42-a548-41cf-9bea-0f2734d74b24\") " pod="ai-tenant-e2e-inf-a-39df6e/test-model-39df6e-kserve-698d88cd99-xgkz7" Sep 10 13:31:18.463097 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:31:18.462786 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/76f7bb42-a548-41cf-9bea-0f2734d74b24-tmp-dir\") pod \"test-model-39df6e-kserve-698d88cd99-xgkz7\" (UID: \"76f7bb42-a548-41cf-9bea-0f2734d74b24\") " pod="ai-tenant-e2e-inf-a-39df6e/test-model-39df6e-kserve-698d88cd99-xgkz7" Sep 10 13:31:18.463241 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:31:18.463223 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xg628\" (UniqueName: \"kubernetes.io/projected/76f7bb42-a548-41cf-9bea-0f2734d74b24-kube-api-access-xg628\") pod \"test-model-39df6e-kserve-698d88cd99-xgkz7\" (UID: \"76f7bb42-a548-41cf-9bea-0f2734d74b24\") " pod="ai-tenant-e2e-inf-a-39df6e/test-model-39df6e-kserve-698d88cd99-xgkz7" Sep 10 13:31:18.463447 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:31:18.463247 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/76f7bb42-a548-41cf-9bea-0f2734d74b24-tmp-dir\") pod \"test-model-39df6e-kserve-698d88cd99-xgkz7\" (UID: \"76f7bb42-a548-41cf-9bea-0f2734d74b24\") " pod="ai-tenant-e2e-inf-a-39df6e/test-model-39df6e-kserve-698d88cd99-xgkz7" Sep 10 13:31:18.464862 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:31:18.464821 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/76f7bb42-a548-41cf-9bea-0f2734d74b24-dshm\") pod \"test-model-39df6e-kserve-698d88cd99-xgkz7\" (UID: \"76f7bb42-a548-41cf-9bea-0f2734d74b24\") " pod="ai-tenant-e2e-inf-a-39df6e/test-model-39df6e-kserve-698d88cd99-xgkz7" Sep 10 13:31:18.471637 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:31:18.471550 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/76f7bb42-a548-41cf-9bea-0f2734d74b24-tls-certs\") pod \"test-model-39df6e-kserve-698d88cd99-xgkz7\" (UID: \"76f7bb42-a548-41cf-9bea-0f2734d74b24\") " pod="ai-tenant-e2e-inf-a-39df6e/test-model-39df6e-kserve-698d88cd99-xgkz7" Sep 10 13:31:18.479852 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:31:18.479817 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xg628\" (UniqueName: \"kubernetes.io/projected/76f7bb42-a548-41cf-9bea-0f2734d74b24-kube-api-access-xg628\") pod \"test-model-39df6e-kserve-698d88cd99-xgkz7\" (UID: \"76f7bb42-a548-41cf-9bea-0f2734d74b24\") " pod="ai-tenant-e2e-inf-a-39df6e/test-model-39df6e-kserve-698d88cd99-xgkz7" Sep 10 13:31:18.533729 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:31:18.533703 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-39df6e/test-model-39df6e-kserve-698d88cd99-xgkz7" Sep 10 13:31:18.843163 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:31:18.843088 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-39df6e/test-model-39df6e-kserve-698d88cd99-xgkz7"] Sep 10 13:31:18.846489 ip-10-0-139-174 kubenswrapper[2557]: W0910 13:31:18.846464 2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod76f7bb42_a548_41cf_9bea_0f2734d74b24.slice/crio-a7e02e56aff7af454a121d1347ec75589a2a431a02cd266be77eaa5704a175c0 WatchSource:0}: Error finding container a7e02e56aff7af454a121d1347ec75589a2a431a02cd266be77eaa5704a175c0: Status 404 returned error can't find the container with id a7e02e56aff7af454a121d1347ec75589a2a431a02cd266be77eaa5704a175c0 Sep 10 13:31:19.749656 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:31:19.749620 2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-300680f3-547d4f8c46-zflds" Sep 10 13:31:19.816345 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:31:19.816309 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-39df6e/test-model-39df6e-kserve-698d88cd99-xgkz7" event={"ID":"76f7bb42-a548-41cf-9bea-0f2734d74b24","Type":"ContainerStarted","Data":"61db300f7ea9a655270b3da030c6f4a3bdf5dacb278c43c30fdbc91a06d685cb"} Sep 10 13:31:19.816540 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:31:19.816357 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-39df6e/test-model-39df6e-kserve-698d88cd99-xgkz7" event={"ID":"76f7bb42-a548-41cf-9bea-0f2734d74b24","Type":"ContainerStarted","Data":"a7e02e56aff7af454a121d1347ec75589a2a431a02cd266be77eaa5704a175c0"} Sep 10 13:31:19.854830 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:31:19.854776 2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-inf-a-39df6e/test-model-39df6e-kserve-698d88cd99-xgkz7" podStartSLOduration=1.85475828 podStartE2EDuration="1.85475828s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 13:31:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 13:31:19.851123043 +0000 UTC m=+840.158117458" watchObservedRunningTime="2026-09-10 13:31:19.85475828 +0000 UTC m=+840.161752693" Sep 10 13:31:23.808795 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:31:23.808760 2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-af59ca-92bdd"] Sep 10 13:31:23.813885 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:31:23.813864 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-af59ca-92bdd" Sep 10 13:31:23.816533 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:31:23.816509 2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Sep 10 13:31:23.816628 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:31:23.816554 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-wtblv\"" Sep 10 13:31:23.826955 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:31:23.826928 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-af59ca-92bdd"] Sep 10 13:31:23.919482 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:31:23.919414 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c5a1512f-5aa5-4a56-ba53-d01e6492f5c6-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-autores-af59ca-92bdd\" (UID: \"c5a1512f-5aa5-4a56-ba53-d01e6492f5c6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-af59ca-92bdd" Sep 10 13:31:24.020734 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:31:24.020699 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c5a1512f-5aa5-4a56-ba53-d01e6492f5c6-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-autores-af59ca-92bdd\" (UID: \"c5a1512f-5aa5-4a56-ba53-d01e6492f5c6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-af59ca-92bdd" Sep 10 13:31:24.021376 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:31:24.021355 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c5a1512f-5aa5-4a56-ba53-d01e6492f5c6-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-autores-af59ca-92bdd\" (UID: \"c5a1512f-5aa5-4a56-ba53-d01e6492f5c6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-af59ca-92bdd" Sep 10 13:31:24.126287 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:31:24.126203 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-af59ca-92bdd" Sep 10 13:31:24.430998 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:31:24.430966 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-af59ca-92bdd"] Sep 10 13:31:24.431333 ip-10-0-139-174 kubenswrapper[2557]: W0910 13:31:24.431309 2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc5a1512f_5aa5_4a56_ba53_d01e6492f5c6.slice/crio-188e85608d256108279018815c4813721bc8878028d7a90aa921ca6ea26f6a3b WatchSource:0}: Error finding container 188e85608d256108279018815c4813721bc8878028d7a90aa921ca6ea26f6a3b: Status 404 returned error can't find the container with id 188e85608d256108279018815c4813721bc8878028d7a90aa921ca6ea26f6a3b Sep 10 13:31:24.853393 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:31:24.853349 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-af59ca-92bdd" event={"ID":"c5a1512f-5aa5-4a56-ba53-d01e6492f5c6","Type":"ContainerStarted","Data":"188e85608d256108279018815c4813721bc8878028d7a90aa921ca6ea26f6a3b"} Sep 10 13:31:25.861831 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:31:25.861797 2557 generic.go:356] "Generic (PLEG): container finished" podID="c5a1512f-5aa5-4a56-ba53-d01e6492f5c6" containerID="aa9cee7d80774c3c2aaf84cc23412eb7d5678043d6d223ca65c1f95dc117a78b" exitCode=0 Sep 10 13:31:25.862167 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:31:25.861892 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-af59ca-92bdd" event={"ID":"c5a1512f-5aa5-4a56-ba53-d01e6492f5c6","Type":"ContainerDied","Data":"aa9cee7d80774c3c2aaf84cc23412eb7d5678043d6d223ca65c1f95dc117a78b"} Sep 10 13:31:27.023924 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:31:27.023892 2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-af59ca-92bdd" Sep 10 13:31:27.053415 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:31:27.053066 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c5a1512f-5aa5-4a56-ba53-d01e6492f5c6-maas-api-service-ca\") pod \"c5a1512f-5aa5-4a56-ba53-d01e6492f5c6\" (UID: \"c5a1512f-5aa5-4a56-ba53-d01e6492f5c6\") " Sep 10 13:31:27.053631 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:31:27.053461 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c5a1512f-5aa5-4a56-ba53-d01e6492f5c6-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "c5a1512f-5aa5-4a56-ba53-d01e6492f5c6" (UID: "c5a1512f-5aa5-4a56-ba53-d01e6492f5c6"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 10 13:31:27.154386 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:31:27.154293 2557 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c5a1512f-5aa5-4a56-ba53-d01e6492f5c6-maas-api-service-ca\") on node \"ip-10-0-139-174.ec2.internal\" DevicePath \"\"" Sep 10 13:31:27.876684 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:31:27.876639 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-af59ca-92bdd" event={"ID":"c5a1512f-5aa5-4a56-ba53-d01e6492f5c6","Type":"ContainerDied","Data":"188e85608d256108279018815c4813721bc8878028d7a90aa921ca6ea26f6a3b"} Sep 10 13:31:27.876684 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:31:27.876662 2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-af59ca-92bdd" Sep 10 13:31:27.876684 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:31:27.876689 2557 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="188e85608d256108279018815c4813721bc8878028d7a90aa921ca6ea26f6a3b" Sep 10 13:31:28.534090 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:31:28.534056 2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-a-39df6e/test-model-39df6e-kserve-698d88cd99-xgkz7" Sep 10 13:31:28.534090 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:31:28.534100 2557 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-a-39df6e/test-model-39df6e-kserve-698d88cd99-xgkz7" Sep 10 13:31:28.547626 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:31:28.547590 2557 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-a-39df6e/test-model-39df6e-kserve-698d88cd99-xgkz7" Sep 10 13:31:28.895497 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:31:28.895470 2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-a-39df6e/test-model-39df6e-kserve-698d88cd99-xgkz7" Sep 10 13:31:29.896581 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:31:29.896531 2557 generic.go:356] "Generic (PLEG): container finished" podID="27bdc7ae-a8d0-43fb-8d70-7a247ef9db39" containerID="95f12407cae6137473e8d381f747c194eb74293e0bb2e45deabefdc156c7f394" exitCode=0 Sep 10 13:31:29.897074 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:31:29.896615 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-300680f3-openshift-default-584b5ff64d-t2ccg" event={"ID":"27bdc7ae-a8d0-43fb-8d70-7a247ef9db39","Type":"ContainerDied","Data":"95f12407cae6137473e8d381f747c194eb74293e0bb2e45deabefdc156c7f394"} Sep 10 13:31:29.897328 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:31:29.897307 2557 scope.go:117] "RemoveContainer" containerID="95f12407cae6137473e8d381f747c194eb74293e0bb2e45deabefdc156c7f394" Sep 10 13:31:29.898758 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:31:29.898722 2557 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892152Ki","pods":"250"} Sep 10 13:31:29.898855 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:31:29.898817 2557 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892152Ki","pods":"250"} Sep 10 13:31:29.898919 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:31:29.898866 2557 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892152Ki","pods":"250"} Sep 10 13:31:30.119598 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:31:30.119566 2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-300680f3-zkh8l"] Sep 10 13:31:30.120005 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:31:30.119990 2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c5a1512f-5aa5-4a56-ba53-d01e6492f5c6" containerName="revoke-keys" Sep 10 13:31:30.120005 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:31:30.120006 2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="c5a1512f-5aa5-4a56-ba53-d01e6492f5c6" containerName="revoke-keys" Sep 10 13:31:30.120152 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:31:30.120088 2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="c5a1512f-5aa5-4a56-ba53-d01e6492f5c6" containerName="revoke-keys" Sep 10 13:31:30.122388 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:31:30.122363 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-300680f3-zkh8l" Sep 10 13:31:30.125069 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:31:30.125048 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-wtblv\"" Sep 10 13:31:30.125195 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:31:30.125108 2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Sep 10 13:31:30.138153 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:31:30.138128 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-300680f3-zkh8l"] Sep 10 13:31:30.186479 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:31:30.186341 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/84befc70-831c-4927-89ff-d019f1127a29-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-300680f3-zkh8l\" (UID: \"84befc70-831c-4927-89ff-d019f1127a29\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-300680f3-zkh8l" Sep 10 13:31:30.287261 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:31:30.287219 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/84befc70-831c-4927-89ff-d019f1127a29-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-300680f3-zkh8l\" (UID: \"84befc70-831c-4927-89ff-d019f1127a29\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-300680f3-zkh8l" Sep 10 13:31:30.288182 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:31:30.287982 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/84befc70-831c-4927-89ff-d019f1127a29-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-300680f3-zkh8l\" (UID: \"84befc70-831c-4927-89ff-d019f1127a29\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-300680f3-zkh8l" Sep 10 13:31:30.434061 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:31:30.434031 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-300680f3-zkh8l" Sep 10 13:31:30.731063 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:31:30.731027 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-300680f3-zkh8l"] Sep 10 13:31:30.733196 ip-10-0-139-174 kubenswrapper[2557]: W0910 13:31:30.733165 2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod84befc70_831c_4927_89ff_d019f1127a29.slice/crio-366f2ddd5483beb92c49e756731824501cb27955080f0a911c21cfbf767b0f7e WatchSource:0}: Error finding container 366f2ddd5483beb92c49e756731824501cb27955080f0a911c21cfbf767b0f7e: Status 404 returned error can't find the container with id 366f2ddd5483beb92c49e756731824501cb27955080f0a911c21cfbf767b0f7e Sep 10 13:31:30.905357 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:31:30.905316 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-300680f3-openshift-default-584b5ff64d-t2ccg" event={"ID":"27bdc7ae-a8d0-43fb-8d70-7a247ef9db39","Type":"ContainerStarted","Data":"9240d00a66f843138ee737f802b6010289c8edd78833979d44cf8c7ca3e49f86"} Sep 10 13:31:30.906649 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:31:30.906624 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-300680f3-zkh8l" event={"ID":"84befc70-831c-4927-89ff-d019f1127a29","Type":"ContainerStarted","Data":"366f2ddd5483beb92c49e756731824501cb27955080f0a911c21cfbf767b0f7e"} Sep 10 13:31:31.021358 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:31:31.021275 2557 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-300680f3-openshift-default-584b5ff64d-t2ccg" Sep 10 13:31:31.022624 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:31:31.022596 2557 patch_prober.go:28] interesting pod/e2e-ait-300680f3-openshift-default-584b5ff64d-t2ccg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.54:15021/healthz/ready\": dial tcp 10.134.0.54:15021: connect: connection refused" start-of-body= Sep 10 13:31:31.022751 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:31:31.022644 2557 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-300680f3-openshift-default-584b5ff64d-t2ccg" podUID="27bdc7ae-a8d0-43fb-8d70-7a247ef9db39" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.54:15021/healthz/ready\": dial tcp 10.134.0.54:15021: connect: connection refused" Sep 10 13:31:31.913564 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:31:31.913528 2557 generic.go:356] "Generic (PLEG): container finished" podID="84befc70-831c-4927-89ff-d019f1127a29" containerID="543e4e1d4708b4cb66ea7b966ca706ee422276f11427d3ab638fbd36c6879cbf" exitCode=0 Sep 10 13:31:31.913952 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:31:31.913621 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-300680f3-zkh8l" event={"ID":"84befc70-831c-4927-89ff-d019f1127a29","Type":"ContainerDied","Data":"543e4e1d4708b4cb66ea7b966ca706ee422276f11427d3ab638fbd36c6879cbf"} Sep 10 13:31:32.021142 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:31:32.021050 2557 patch_prober.go:28] interesting pod/e2e-ait-300680f3-openshift-default-584b5ff64d-t2ccg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.54:15021/healthz/ready\": dial tcp 10.134.0.54:15021: connect: connection refused" start-of-body= Sep 10 13:31:32.021142 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:31:32.021103 2557 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-300680f3-openshift-default-584b5ff64d-t2ccg" podUID="27bdc7ae-a8d0-43fb-8d70-7a247ef9db39" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.54:15021/healthz/ready\": dial tcp 10.134.0.54:15021: connect: connection refused" Sep 10 13:31:33.021631 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:31:33.021596 2557 patch_prober.go:28] interesting pod/e2e-ait-300680f3-openshift-default-584b5ff64d-t2ccg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.54:15021/healthz/ready\": dial tcp 10.134.0.54:15021: connect: connection refused" start-of-body= Sep 10 13:31:33.021951 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:31:33.021669 2557 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-300680f3-openshift-default-584b5ff64d-t2ccg" podUID="27bdc7ae-a8d0-43fb-8d70-7a247ef9db39" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.54:15021/healthz/ready\": dial tcp 10.134.0.54:15021: connect: connection refused" Sep 10 13:31:33.087598 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:31:33.087572 2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-300680f3-zkh8l" Sep 10 13:31:33.117929 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:31:33.117904 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/84befc70-831c-4927-89ff-d019f1127a29-maas-api-service-ca\") pod \"84befc70-831c-4927-89ff-d019f1127a29\" (UID: \"84befc70-831c-4927-89ff-d019f1127a29\") " Sep 10 13:31:33.118279 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:31:33.118244 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/84befc70-831c-4927-89ff-d019f1127a29-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "84befc70-831c-4927-89ff-d019f1127a29" (UID: "84befc70-831c-4927-89ff-d019f1127a29"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 10 13:31:33.219205 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:31:33.219174 2557 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/84befc70-831c-4927-89ff-d019f1127a29-maas-api-service-ca\") on node \"ip-10-0-139-174.ec2.internal\" DevicePath \"\"" Sep 10 13:31:33.709280 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:31:33.709233 2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-300680f3-openshift-default-584b5ff64d-t2ccg" Sep 10 13:31:33.927321 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:31:33.927291 2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-300680f3-zkh8l" Sep 10 13:31:33.927509 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:31:33.927291 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-300680f3-zkh8l" event={"ID":"84befc70-831c-4927-89ff-d019f1127a29","Type":"ContainerDied","Data":"366f2ddd5483beb92c49e756731824501cb27955080f0a911c21cfbf767b0f7e"} Sep 10 13:31:33.927509 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:31:33.927414 2557 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="366f2ddd5483beb92c49e756731824501cb27955080f0a911c21cfbf767b0f7e" Sep 10 13:31:34.021491 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:31:34.021375 2557 patch_prober.go:28] interesting pod/e2e-ait-300680f3-openshift-default-584b5ff64d-t2ccg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.54:15021/healthz/ready\": dial tcp 10.134.0.54:15021: connect: connection refused" start-of-body= Sep 10 13:31:34.021491 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:31:34.021448 2557 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-300680f3-openshift-default-584b5ff64d-t2ccg" podUID="27bdc7ae-a8d0-43fb-8d70-7a247ef9db39" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.54:15021/healthz/ready\": dial tcp 10.134.0.54:15021: connect: connection refused" Sep 10 13:31:35.020690 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:31:35.020649 2557 patch_prober.go:28] interesting pod/e2e-ait-300680f3-openshift-default-584b5ff64d-t2ccg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.54:15021/healthz/ready\": dial tcp 10.134.0.54:15021: connect: connection refused" start-of-body= Sep 10 13:31:35.021056 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:31:35.020710 2557 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-300680f3-openshift-default-584b5ff64d-t2ccg" podUID="27bdc7ae-a8d0-43fb-8d70-7a247ef9db39" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.54:15021/healthz/ready\": dial tcp 10.134.0.54:15021: connect: connection refused" Sep 10 13:31:35.166087 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:31:35.166044 2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-autores-af59ca-29817450-4zl7r"] Sep 10 13:31:35.170210 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:31:35.170181 2557 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-autores-af59ca-29817450-4zl7r"] Sep 10 13:31:35.569457 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:31:35.567505 2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-af59ca-92bdd"] Sep 10 13:31:35.570103 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:31:35.570063 2557 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-af59ca-92bdd"] Sep 10 13:31:36.020833 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:31:36.020792 2557 patch_prober.go:28] interesting pod/e2e-ait-300680f3-openshift-default-584b5ff64d-t2ccg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.54:15021/healthz/ready\": dial tcp 10.134.0.54:15021: connect: connection refused" start-of-body= Sep 10 13:31:36.021229 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:31:36.020864 2557 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-300680f3-openshift-default-584b5ff64d-t2ccg" podUID="27bdc7ae-a8d0-43fb-8d70-7a247ef9db39" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.54:15021/healthz/ready\": dial tcp 10.134.0.54:15021: connect: connection refused" Sep 10 13:31:36.220825 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:31:36.220796 2557 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c5a1512f-5aa5-4a56-ba53-d01e6492f5c6" path="/var/lib/kubelet/pods/c5a1512f-5aa5-4a56-ba53-d01e6492f5c6/volumes" Sep 10 13:31:36.221101 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:31:36.221088 2557 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c77626f4-aa4b-49b9-8a28-fb070afa5c26" path="/var/lib/kubelet/pods/c77626f4-aa4b-49b9-8a28-fb070afa5c26/volumes" Sep 10 13:31:36.746545 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:31:36.746512 2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-300680f3-547d4f8c46-zflds"] Sep 10 13:31:36.746753 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:31:36.746729 2557 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ait-300680f3-547d4f8c46-zflds" podUID="224fe991-782f-4f11-a321-3e6c4f03c0b8" containerName="payload-pre-processing" containerID="cri-o://982f36cce16297922720f34c5be079b861fde286f0e58cd6509a67b8e5890c32" gracePeriod=30 Sep 10 13:31:36.909586 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:31:36.909151 2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-300680f3-zkh8l"] Sep 10 13:31:36.912668 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:31:36.912645 2557 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-300680f3-zkh8l"] Sep 10 13:31:36.950104 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:31:36.950073 2557 generic.go:356] "Generic (PLEG): container finished" podID="224fe991-782f-4f11-a321-3e6c4f03c0b8" containerID="982f36cce16297922720f34c5be079b861fde286f0e58cd6509a67b8e5890c32" exitCode=0 Sep 10 13:31:36.950251 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:31:36.950151 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-300680f3-547d4f8c46-zflds" event={"ID":"224fe991-782f-4f11-a321-3e6c4f03c0b8","Type":"ContainerDied","Data":"982f36cce16297922720f34c5be079b861fde286f0e58cd6509a67b8e5890c32"} Sep 10 13:31:37.004130 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:31:37.004069 2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-300680f3-547d4f8c46-zflds" Sep 10 13:31:37.024744 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:31:37.024609 2557 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-300680f3-openshift-default-584b5ff64d-t2ccg" Sep 10 13:31:37.026591 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:31:37.026569 2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-300680f3-openshift-default-584b5ff64d-t2ccg" Sep 10 13:31:37.072626 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:31:37.072586 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-m65gt\" (UniqueName: \"kubernetes.io/projected/224fe991-782f-4f11-a321-3e6c4f03c0b8-kube-api-access-m65gt\") pod \"224fe991-782f-4f11-a321-3e6c4f03c0b8\" (UID: \"224fe991-782f-4f11-a321-3e6c4f03c0b8\") " Sep 10 13:31:37.072965 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:31:37.072941 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/224fe991-782f-4f11-a321-3e6c4f03c0b8-plugins-config-volume\") pod \"224fe991-782f-4f11-a321-3e6c4f03c0b8\" (UID: \"224fe991-782f-4f11-a321-3e6c4f03c0b8\") " Sep 10 13:31:37.074022 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:31:37.073998 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/224fe991-782f-4f11-a321-3e6c4f03c0b8-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "224fe991-782f-4f11-a321-3e6c4f03c0b8" (UID: "224fe991-782f-4f11-a321-3e6c4f03c0b8"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 10 13:31:37.076250 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:31:37.076217 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/224fe991-782f-4f11-a321-3e6c4f03c0b8-kube-api-access-m65gt" (OuterVolumeSpecName: "kube-api-access-m65gt") pod "224fe991-782f-4f11-a321-3e6c4f03c0b8" (UID: "224fe991-782f-4f11-a321-3e6c4f03c0b8"). InnerVolumeSpecName "kube-api-access-m65gt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 10 13:31:37.174252 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:31:37.174212 2557 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m65gt\" (UniqueName: \"kubernetes.io/projected/224fe991-782f-4f11-a321-3e6c4f03c0b8-kube-api-access-m65gt\") on node \"ip-10-0-139-174.ec2.internal\" DevicePath \"\"" Sep 10 13:31:37.174252 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:31:37.174247 2557 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/224fe991-782f-4f11-a321-3e6c4f03c0b8-plugins-config-volume\") on node \"ip-10-0-139-174.ec2.internal\" DevicePath \"\"" Sep 10 13:31:37.957502 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:31:37.957472 2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-300680f3-547d4f8c46-zflds" Sep 10 13:31:37.957704 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:31:37.957517 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-300680f3-547d4f8c46-zflds" event={"ID":"224fe991-782f-4f11-a321-3e6c4f03c0b8","Type":"ContainerDied","Data":"052643ad843956db0d5ac7351dd54183e0e68e82442ed6f5017d962e425430bf"} Sep 10 13:31:37.957704 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:31:37.957579 2557 scope.go:117] "RemoveContainer" containerID="982f36cce16297922720f34c5be079b861fde286f0e58cd6509a67b8e5890c32" Sep 10 13:31:37.980477 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:31:37.980447 2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-300680f3-547d4f8c46-zflds"] Sep 10 13:31:37.984996 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:31:37.984975 2557 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-300680f3-547d4f8c46-zflds"] Sep 10 13:31:38.128679 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:31:38.128641 2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-300680f3-openshift-default-584b5ff64d-t2ccg"] Sep 10 13:31:38.224686 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:31:38.224591 2557 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="224fe991-782f-4f11-a321-3e6c4f03c0b8" path="/var/lib/kubelet/pods/224fe991-782f-4f11-a321-3e6c4f03c0b8/volumes" Sep 10 13:31:38.225127 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:31:38.225107 2557 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="84befc70-831c-4927-89ff-d019f1127a29" path="/var/lib/kubelet/pods/84befc70-831c-4927-89ff-d019f1127a29/volumes" Sep 10 13:31:38.965947 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:31:38.965913 2557 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-ait-300680f3-openshift-default-584b5ff64d-t2ccg" podUID="27bdc7ae-a8d0-43fb-8d70-7a247ef9db39" containerName="istio-proxy" containerID="cri-o://9240d00a66f843138ee737f802b6010289c8edd78833979d44cf8c7ca3e49f86" gracePeriod=30 Sep 10 13:31:44.230371 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:31:44.230308 2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-300680f3-openshift-default-584b5ff64d-t2ccg" Sep 10 13:31:44.348023 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:31:44.347995 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/27bdc7ae-a8d0-43fb-8d70-7a247ef9db39-workload-socket\") pod \"27bdc7ae-a8d0-43fb-8d70-7a247ef9db39\" (UID: \"27bdc7ae-a8d0-43fb-8d70-7a247ef9db39\") " Sep 10 13:31:44.348023 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:31:44.348030 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/27bdc7ae-a8d0-43fb-8d70-7a247ef9db39-istio-token\") pod \"27bdc7ae-a8d0-43fb-8d70-7a247ef9db39\" (UID: \"27bdc7ae-a8d0-43fb-8d70-7a247ef9db39\") " Sep 10 13:31:44.348237 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:31:44.348103 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/27bdc7ae-a8d0-43fb-8d70-7a247ef9db39-istio-podinfo\") pod \"27bdc7ae-a8d0-43fb-8d70-7a247ef9db39\" (UID: \"27bdc7ae-a8d0-43fb-8d70-7a247ef9db39\") " Sep 10 13:31:44.348237 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:31:44.348120 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-cqfv4\" (UniqueName: \"kubernetes.io/projected/27bdc7ae-a8d0-43fb-8d70-7a247ef9db39-kube-api-access-cqfv4\") pod \"27bdc7ae-a8d0-43fb-8d70-7a247ef9db39\" (UID: \"27bdc7ae-a8d0-43fb-8d70-7a247ef9db39\") " Sep 10 13:31:44.348237 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:31:44.348140 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/27bdc7ae-a8d0-43fb-8d70-7a247ef9db39-istiod-ca-cert\") pod \"27bdc7ae-a8d0-43fb-8d70-7a247ef9db39\" (UID: \"27bdc7ae-a8d0-43fb-8d70-7a247ef9db39\") " Sep 10 13:31:44.348237 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:31:44.348158 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/27bdc7ae-a8d0-43fb-8d70-7a247ef9db39-workload-certs\") pod \"27bdc7ae-a8d0-43fb-8d70-7a247ef9db39\" (UID: \"27bdc7ae-a8d0-43fb-8d70-7a247ef9db39\") " Sep 10 13:31:44.348237 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:31:44.348231 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/27bdc7ae-a8d0-43fb-8d70-7a247ef9db39-credential-socket\") pod \"27bdc7ae-a8d0-43fb-8d70-7a247ef9db39\" (UID: \"27bdc7ae-a8d0-43fb-8d70-7a247ef9db39\") " Sep 10 13:31:44.348520 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:31:44.348264 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/27bdc7ae-a8d0-43fb-8d70-7a247ef9db39-istio-envoy\") pod \"27bdc7ae-a8d0-43fb-8d70-7a247ef9db39\" (UID: \"27bdc7ae-a8d0-43fb-8d70-7a247ef9db39\") " Sep 10 13:31:44.348520 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:31:44.348294 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/27bdc7ae-a8d0-43fb-8d70-7a247ef9db39-istio-data\") pod \"27bdc7ae-a8d0-43fb-8d70-7a247ef9db39\" (UID: \"27bdc7ae-a8d0-43fb-8d70-7a247ef9db39\") " Sep 10 13:31:44.348520 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:31:44.348474 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/27bdc7ae-a8d0-43fb-8d70-7a247ef9db39-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "27bdc7ae-a8d0-43fb-8d70-7a247ef9db39" (UID: "27bdc7ae-a8d0-43fb-8d70-7a247ef9db39"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 13:31:44.348676 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:31:44.348557 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/27bdc7ae-a8d0-43fb-8d70-7a247ef9db39-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "27bdc7ae-a8d0-43fb-8d70-7a247ef9db39" (UID: "27bdc7ae-a8d0-43fb-8d70-7a247ef9db39"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 13:31:44.348676 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:31:44.348640 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/27bdc7ae-a8d0-43fb-8d70-7a247ef9db39-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "27bdc7ae-a8d0-43fb-8d70-7a247ef9db39" (UID: "27bdc7ae-a8d0-43fb-8d70-7a247ef9db39"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 10 13:31:44.348789 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:31:44.348277 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/27bdc7ae-a8d0-43fb-8d70-7a247ef9db39-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "27bdc7ae-a8d0-43fb-8d70-7a247ef9db39" (UID: "27bdc7ae-a8d0-43fb-8d70-7a247ef9db39"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 13:31:44.348847 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:31:44.348781 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/27bdc7ae-a8d0-43fb-8d70-7a247ef9db39-istio-data" (OuterVolumeSpecName: "istio-data") pod "27bdc7ae-a8d0-43fb-8d70-7a247ef9db39" (UID: "27bdc7ae-a8d0-43fb-8d70-7a247ef9db39"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 13:31:44.348997 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:31:44.348980 2557 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/27bdc7ae-a8d0-43fb-8d70-7a247ef9db39-workload-socket\") on node \"ip-10-0-139-174.ec2.internal\" DevicePath \"\"" Sep 10 13:31:44.349045 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:31:44.349003 2557 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/27bdc7ae-a8d0-43fb-8d70-7a247ef9db39-istiod-ca-cert\") on node \"ip-10-0-139-174.ec2.internal\" DevicePath \"\"" Sep 10 13:31:44.349045 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:31:44.349017 2557 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/27bdc7ae-a8d0-43fb-8d70-7a247ef9db39-workload-certs\") on node \"ip-10-0-139-174.ec2.internal\" DevicePath \"\"" Sep 10 13:31:44.349045 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:31:44.349030 2557 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/27bdc7ae-a8d0-43fb-8d70-7a247ef9db39-credential-socket\") on node \"ip-10-0-139-174.ec2.internal\" DevicePath \"\"" Sep 10 13:31:44.350434 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:31:44.350391 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/27bdc7ae-a8d0-43fb-8d70-7a247ef9db39-istio-token" (OuterVolumeSpecName: "istio-token") pod "27bdc7ae-a8d0-43fb-8d70-7a247ef9db39" (UID: "27bdc7ae-a8d0-43fb-8d70-7a247ef9db39"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 10 13:31:44.350633 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:31:44.350610 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/27bdc7ae-a8d0-43fb-8d70-7a247ef9db39-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "27bdc7ae-a8d0-43fb-8d70-7a247ef9db39" (UID: "27bdc7ae-a8d0-43fb-8d70-7a247ef9db39"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 13:31:44.350904 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:31:44.350878 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/27bdc7ae-a8d0-43fb-8d70-7a247ef9db39-kube-api-access-cqfv4" (OuterVolumeSpecName: "kube-api-access-cqfv4") pod "27bdc7ae-a8d0-43fb-8d70-7a247ef9db39" (UID: "27bdc7ae-a8d0-43fb-8d70-7a247ef9db39"). InnerVolumeSpecName "kube-api-access-cqfv4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 10 13:31:44.350983 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:31:44.350961 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/27bdc7ae-a8d0-43fb-8d70-7a247ef9db39-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "27bdc7ae-a8d0-43fb-8d70-7a247ef9db39" (UID: "27bdc7ae-a8d0-43fb-8d70-7a247ef9db39"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Sep 10 13:31:44.450583 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:31:44.450560 2557 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/27bdc7ae-a8d0-43fb-8d70-7a247ef9db39-istio-podinfo\") on node \"ip-10-0-139-174.ec2.internal\" DevicePath \"\"" Sep 10 13:31:44.450583 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:31:44.450582 2557 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cqfv4\" (UniqueName: \"kubernetes.io/projected/27bdc7ae-a8d0-43fb-8d70-7a247ef9db39-kube-api-access-cqfv4\") on node \"ip-10-0-139-174.ec2.internal\" DevicePath \"\"" Sep 10 13:31:44.450725 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:31:44.450591 2557 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/27bdc7ae-a8d0-43fb-8d70-7a247ef9db39-istio-envoy\") on node \"ip-10-0-139-174.ec2.internal\" DevicePath \"\"" Sep 10 13:31:44.450725 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:31:44.450600 2557 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/27bdc7ae-a8d0-43fb-8d70-7a247ef9db39-istio-data\") on node \"ip-10-0-139-174.ec2.internal\" DevicePath \"\"" Sep 10 13:31:44.450725 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:31:44.450607 2557 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/27bdc7ae-a8d0-43fb-8d70-7a247ef9db39-istio-token\") on node \"ip-10-0-139-174.ec2.internal\" DevicePath \"\"" Sep 10 13:31:45.007395 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:31:45.007362 2557 generic.go:356] "Generic (PLEG): container finished" podID="27bdc7ae-a8d0-43fb-8d70-7a247ef9db39" containerID="9240d00a66f843138ee737f802b6010289c8edd78833979d44cf8c7ca3e49f86" exitCode=0 Sep 10 13:31:45.007583 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:31:45.007443 2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-300680f3-openshift-default-584b5ff64d-t2ccg" Sep 10 13:31:45.007583 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:31:45.007449 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-300680f3-openshift-default-584b5ff64d-t2ccg" event={"ID":"27bdc7ae-a8d0-43fb-8d70-7a247ef9db39","Type":"ContainerDied","Data":"9240d00a66f843138ee737f802b6010289c8edd78833979d44cf8c7ca3e49f86"} Sep 10 13:31:45.007583 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:31:45.007510 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-300680f3-openshift-default-584b5ff64d-t2ccg" event={"ID":"27bdc7ae-a8d0-43fb-8d70-7a247ef9db39","Type":"ContainerDied","Data":"c9651c917370cb19a84ba48a4656d670d1871375fdca238799b0344d8a1a9d2a"} Sep 10 13:31:45.007583 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:31:45.007534 2557 scope.go:117] "RemoveContainer" containerID="9240d00a66f843138ee737f802b6010289c8edd78833979d44cf8c7ca3e49f86" Sep 10 13:31:45.018171 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:31:45.018154 2557 scope.go:117] "RemoveContainer" containerID="95f12407cae6137473e8d381f747c194eb74293e0bb2e45deabefdc156c7f394" Sep 10 13:31:45.027347 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:31:45.027327 2557 scope.go:117] "RemoveContainer" containerID="9240d00a66f843138ee737f802b6010289c8edd78833979d44cf8c7ca3e49f86" Sep 10 13:31:45.027632 ip-10-0-139-174 kubenswrapper[2557]: E0910 13:31:45.027614 2557 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9240d00a66f843138ee737f802b6010289c8edd78833979d44cf8c7ca3e49f86\": container with ID starting with 9240d00a66f843138ee737f802b6010289c8edd78833979d44cf8c7ca3e49f86 not found: ID does not exist" containerID="9240d00a66f843138ee737f802b6010289c8edd78833979d44cf8c7ca3e49f86" Sep 10 13:31:45.027683 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:31:45.027641 2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9240d00a66f843138ee737f802b6010289c8edd78833979d44cf8c7ca3e49f86"} err="failed to get container status \"9240d00a66f843138ee737f802b6010289c8edd78833979d44cf8c7ca3e49f86\": rpc error: code = NotFound desc = could not find container \"9240d00a66f843138ee737f802b6010289c8edd78833979d44cf8c7ca3e49f86\": container with ID starting with 9240d00a66f843138ee737f802b6010289c8edd78833979d44cf8c7ca3e49f86 not found: ID does not exist" Sep 10 13:31:45.027683 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:31:45.027657 2557 scope.go:117] "RemoveContainer" containerID="95f12407cae6137473e8d381f747c194eb74293e0bb2e45deabefdc156c7f394" Sep 10 13:31:45.027934 ip-10-0-139-174 kubenswrapper[2557]: E0910 13:31:45.027915 2557 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"95f12407cae6137473e8d381f747c194eb74293e0bb2e45deabefdc156c7f394\": container with ID starting with 95f12407cae6137473e8d381f747c194eb74293e0bb2e45deabefdc156c7f394 not found: ID does not exist" containerID="95f12407cae6137473e8d381f747c194eb74293e0bb2e45deabefdc156c7f394" Sep 10 13:31:45.027986 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:31:45.027939 2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"95f12407cae6137473e8d381f747c194eb74293e0bb2e45deabefdc156c7f394"} err="failed to get container status \"95f12407cae6137473e8d381f747c194eb74293e0bb2e45deabefdc156c7f394\": rpc error: code = NotFound desc = could not find container \"95f12407cae6137473e8d381f747c194eb74293e0bb2e45deabefdc156c7f394\": container with ID starting with 95f12407cae6137473e8d381f747c194eb74293e0bb2e45deabefdc156c7f394 not found: ID does not exist" Sep 10 13:31:45.052598 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:31:45.052575 2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-300680f3-openshift-default-584b5ff64d-t2ccg"] Sep 10 13:31:45.061866 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:31:45.061844 2557 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ait-300680f3-openshift-default-584b5ff64d-t2ccg"] Sep 10 13:31:45.741883 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:31:45.741852 2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-autores-af59ca-openshift-default-887c6d76d-t4qb9"] Sep 10 13:31:45.742364 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:31:45.742165 2557 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-autores-af59ca-openshift-default-887c6d76d-t4qb9" podUID="22c3a6a3-69e4-43db-8768-525668cb1928" containerName="istio-proxy" containerID="cri-o://caffa4bde3e2aecd434145033c8a8291c60306c41704f29f98bed8fb7fea89bb" gracePeriod=30 Sep 10 13:31:46.221049 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:31:46.221009 2557 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="27bdc7ae-a8d0-43fb-8d70-7a247ef9db39" path="/var/lib/kubelet/pods/27bdc7ae-a8d0-43fb-8d70-7a247ef9db39/volumes" Sep 10 13:31:46.641762 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:31:46.641684 2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-explicit-a26db0-openshift-default-74c74695cb-rpxgt"] Sep 10 13:31:46.642141 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:31:46.642127 2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="224fe991-782f-4f11-a321-3e6c4f03c0b8" containerName="payload-pre-processing" Sep 10 13:31:46.642199 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:31:46.642146 2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="224fe991-782f-4f11-a321-3e6c4f03c0b8" containerName="payload-pre-processing" Sep 10 13:31:46.642199 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:31:46.642176 2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="84befc70-831c-4927-89ff-d019f1127a29" containerName="revoke-keys" Sep 10 13:31:46.642199 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:31:46.642182 2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="84befc70-831c-4927-89ff-d019f1127a29" containerName="revoke-keys" Sep 10 13:31:46.642199 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:31:46.642192 2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="27bdc7ae-a8d0-43fb-8d70-7a247ef9db39" containerName="istio-proxy" Sep 10 13:31:46.642199 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:31:46.642197 2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="27bdc7ae-a8d0-43fb-8d70-7a247ef9db39" containerName="istio-proxy" Sep 10 13:31:46.642370 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:31:46.642205 2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="27bdc7ae-a8d0-43fb-8d70-7a247ef9db39" containerName="istio-proxy" Sep 10 13:31:46.642370 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:31:46.642210 2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="27bdc7ae-a8d0-43fb-8d70-7a247ef9db39" containerName="istio-proxy" Sep 10 13:31:46.642370 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:31:46.642287 2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="27bdc7ae-a8d0-43fb-8d70-7a247ef9db39" containerName="istio-proxy" Sep 10 13:31:46.642370 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:31:46.642296 2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="224fe991-782f-4f11-a321-3e6c4f03c0b8" containerName="payload-pre-processing" Sep 10 13:31:46.642370 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:31:46.642302 2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="84befc70-831c-4927-89ff-d019f1127a29" containerName="revoke-keys" Sep 10 13:31:46.646849 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:31:46.646830 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-explicit-a26db0-openshift-default-74c74695cb-rpxgt" Sep 10 13:31:46.649776 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:31:46.649755 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-explicit-a26db0-openshift-default-dockercfg-xnbvx\"" Sep 10 13:31:46.688922 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:31:46.688890 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-explicit-a26db0-openshift-default-74c74695cb-rpxgt"] Sep 10 13:31:46.735403 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:31:46.735360 2557 patch_prober.go:28] interesting pod/e2e-autores-af59ca-openshift-default-887c6d76d-t4qb9 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Sep 10 13:31:46.735558 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:31:46.735454 2557 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-autores-af59ca-openshift-default-887c6d76d-t4qb9" podUID="22c3a6a3-69e4-43db-8768-525668cb1928" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Sep 10 13:31:46.774821 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:31:46.774787 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2a5dff35-e18a-44cc-a1a9-ccdc505d818d-istio-podinfo\") pod \"e2e-explicit-a26db0-openshift-default-74c74695cb-rpxgt\" (UID: \"2a5dff35-e18a-44cc-a1a9-ccdc505d818d\") " pod="openshift-ingress/e2e-explicit-a26db0-openshift-default-74c74695cb-rpxgt" Sep 10 13:31:46.775310 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:31:46.774847 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2a5dff35-e18a-44cc-a1a9-ccdc505d818d-credential-socket\") pod \"e2e-explicit-a26db0-openshift-default-74c74695cb-rpxgt\" (UID: \"2a5dff35-e18a-44cc-a1a9-ccdc505d818d\") " pod="openshift-ingress/e2e-explicit-a26db0-openshift-default-74c74695cb-rpxgt" Sep 10 13:31:46.775310 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:31:46.774911 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ppccs\" (UniqueName: \"kubernetes.io/projected/2a5dff35-e18a-44cc-a1a9-ccdc505d818d-kube-api-access-ppccs\") pod \"e2e-explicit-a26db0-openshift-default-74c74695cb-rpxgt\" (UID: \"2a5dff35-e18a-44cc-a1a9-ccdc505d818d\") " pod="openshift-ingress/e2e-explicit-a26db0-openshift-default-74c74695cb-rpxgt" Sep 10 13:31:46.775310 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:31:46.775043 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2a5dff35-e18a-44cc-a1a9-ccdc505d818d-istio-envoy\") pod \"e2e-explicit-a26db0-openshift-default-74c74695cb-rpxgt\" (UID: \"2a5dff35-e18a-44cc-a1a9-ccdc505d818d\") " pod="openshift-ingress/e2e-explicit-a26db0-openshift-default-74c74695cb-rpxgt" Sep 10 13:31:46.775310 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:31:46.775089 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2a5dff35-e18a-44cc-a1a9-ccdc505d818d-istio-token\") pod \"e2e-explicit-a26db0-openshift-default-74c74695cb-rpxgt\" (UID: \"2a5dff35-e18a-44cc-a1a9-ccdc505d818d\") " pod="openshift-ingress/e2e-explicit-a26db0-openshift-default-74c74695cb-rpxgt" Sep 10 13:31:46.775310 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:31:46.775150 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2a5dff35-e18a-44cc-a1a9-ccdc505d818d-workload-socket\") pod \"e2e-explicit-a26db0-openshift-default-74c74695cb-rpxgt\" (UID: \"2a5dff35-e18a-44cc-a1a9-ccdc505d818d\") " pod="openshift-ingress/e2e-explicit-a26db0-openshift-default-74c74695cb-rpxgt" Sep 10 13:31:46.775310 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:31:46.775194 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2a5dff35-e18a-44cc-a1a9-ccdc505d818d-workload-certs\") pod \"e2e-explicit-a26db0-openshift-default-74c74695cb-rpxgt\" (UID: \"2a5dff35-e18a-44cc-a1a9-ccdc505d818d\") " pod="openshift-ingress/e2e-explicit-a26db0-openshift-default-74c74695cb-rpxgt" Sep 10 13:31:46.775310 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:31:46.775253 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2a5dff35-e18a-44cc-a1a9-ccdc505d818d-istio-data\") pod \"e2e-explicit-a26db0-openshift-default-74c74695cb-rpxgt\" (UID: \"2a5dff35-e18a-44cc-a1a9-ccdc505d818d\") " pod="openshift-ingress/e2e-explicit-a26db0-openshift-default-74c74695cb-rpxgt" Sep 10 13:31:46.775310 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:31:46.775282 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2a5dff35-e18a-44cc-a1a9-ccdc505d818d-istiod-ca-cert\") pod \"e2e-explicit-a26db0-openshift-default-74c74695cb-rpxgt\" (UID: \"2a5dff35-e18a-44cc-a1a9-ccdc505d818d\") " pod="openshift-ingress/e2e-explicit-a26db0-openshift-default-74c74695cb-rpxgt" Sep 10 13:31:46.876299 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:31:46.876269 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2a5dff35-e18a-44cc-a1a9-ccdc505d818d-istio-envoy\") pod \"e2e-explicit-a26db0-openshift-default-74c74695cb-rpxgt\" (UID: \"2a5dff35-e18a-44cc-a1a9-ccdc505d818d\") " pod="openshift-ingress/e2e-explicit-a26db0-openshift-default-74c74695cb-rpxgt" Sep 10 13:31:46.876489 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:31:46.876306 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2a5dff35-e18a-44cc-a1a9-ccdc505d818d-istio-token\") pod \"e2e-explicit-a26db0-openshift-default-74c74695cb-rpxgt\" (UID: \"2a5dff35-e18a-44cc-a1a9-ccdc505d818d\") " pod="openshift-ingress/e2e-explicit-a26db0-openshift-default-74c74695cb-rpxgt" Sep 10 13:31:46.876489 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:31:46.876338 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2a5dff35-e18a-44cc-a1a9-ccdc505d818d-workload-socket\") pod \"e2e-explicit-a26db0-openshift-default-74c74695cb-rpxgt\" (UID: \"2a5dff35-e18a-44cc-a1a9-ccdc505d818d\") " pod="openshift-ingress/e2e-explicit-a26db0-openshift-default-74c74695cb-rpxgt" Sep 10 13:31:46.876489 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:31:46.876354 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2a5dff35-e18a-44cc-a1a9-ccdc505d818d-workload-certs\") pod \"e2e-explicit-a26db0-openshift-default-74c74695cb-rpxgt\" (UID: \"2a5dff35-e18a-44cc-a1a9-ccdc505d818d\") " pod="openshift-ingress/e2e-explicit-a26db0-openshift-default-74c74695cb-rpxgt" Sep 10 13:31:46.876489 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:31:46.876384 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2a5dff35-e18a-44cc-a1a9-ccdc505d818d-istio-data\") pod \"e2e-explicit-a26db0-openshift-default-74c74695cb-rpxgt\" (UID: \"2a5dff35-e18a-44cc-a1a9-ccdc505d818d\") " pod="openshift-ingress/e2e-explicit-a26db0-openshift-default-74c74695cb-rpxgt" Sep 10 13:31:46.876489 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:31:46.876401 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2a5dff35-e18a-44cc-a1a9-ccdc505d818d-istiod-ca-cert\") pod \"e2e-explicit-a26db0-openshift-default-74c74695cb-rpxgt\" (UID: \"2a5dff35-e18a-44cc-a1a9-ccdc505d818d\") " pod="openshift-ingress/e2e-explicit-a26db0-openshift-default-74c74695cb-rpxgt" Sep 10 13:31:46.876788 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:31:46.876587 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2a5dff35-e18a-44cc-a1a9-ccdc505d818d-istio-podinfo\") pod \"e2e-explicit-a26db0-openshift-default-74c74695cb-rpxgt\" (UID: \"2a5dff35-e18a-44cc-a1a9-ccdc505d818d\") " pod="openshift-ingress/e2e-explicit-a26db0-openshift-default-74c74695cb-rpxgt" Sep 10 13:31:46.876788 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:31:46.876654 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2a5dff35-e18a-44cc-a1a9-ccdc505d818d-credential-socket\") pod \"e2e-explicit-a26db0-openshift-default-74c74695cb-rpxgt\" (UID: \"2a5dff35-e18a-44cc-a1a9-ccdc505d818d\") " pod="openshift-ingress/e2e-explicit-a26db0-openshift-default-74c74695cb-rpxgt" Sep 10 13:31:46.876788 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:31:46.876686 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ppccs\" (UniqueName: \"kubernetes.io/projected/2a5dff35-e18a-44cc-a1a9-ccdc505d818d-kube-api-access-ppccs\") pod \"e2e-explicit-a26db0-openshift-default-74c74695cb-rpxgt\" (UID: \"2a5dff35-e18a-44cc-a1a9-ccdc505d818d\") " pod="openshift-ingress/e2e-explicit-a26db0-openshift-default-74c74695cb-rpxgt" Sep 10 13:31:46.876936 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:31:46.876813 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2a5dff35-e18a-44cc-a1a9-ccdc505d818d-istio-data\") pod \"e2e-explicit-a26db0-openshift-default-74c74695cb-rpxgt\" (UID: \"2a5dff35-e18a-44cc-a1a9-ccdc505d818d\") " pod="openshift-ingress/e2e-explicit-a26db0-openshift-default-74c74695cb-rpxgt" Sep 10 13:31:46.876936 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:31:46.876871 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2a5dff35-e18a-44cc-a1a9-ccdc505d818d-workload-socket\") pod \"e2e-explicit-a26db0-openshift-default-74c74695cb-rpxgt\" (UID: \"2a5dff35-e18a-44cc-a1a9-ccdc505d818d\") " pod="openshift-ingress/e2e-explicit-a26db0-openshift-default-74c74695cb-rpxgt" Sep 10 13:31:46.877069 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:31:46.877039 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2a5dff35-e18a-44cc-a1a9-ccdc505d818d-workload-certs\") pod \"e2e-explicit-a26db0-openshift-default-74c74695cb-rpxgt\" (UID: \"2a5dff35-e18a-44cc-a1a9-ccdc505d818d\") " pod="openshift-ingress/e2e-explicit-a26db0-openshift-default-74c74695cb-rpxgt" Sep 10 13:31:46.877134 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:31:46.877104 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2a5dff35-e18a-44cc-a1a9-ccdc505d818d-credential-socket\") pod \"e2e-explicit-a26db0-openshift-default-74c74695cb-rpxgt\" (UID: \"2a5dff35-e18a-44cc-a1a9-ccdc505d818d\") " pod="openshift-ingress/e2e-explicit-a26db0-openshift-default-74c74695cb-rpxgt" Sep 10 13:31:46.877179 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:31:46.877142 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2a5dff35-e18a-44cc-a1a9-ccdc505d818d-istiod-ca-cert\") pod \"e2e-explicit-a26db0-openshift-default-74c74695cb-rpxgt\" (UID: \"2a5dff35-e18a-44cc-a1a9-ccdc505d818d\") " pod="openshift-ingress/e2e-explicit-a26db0-openshift-default-74c74695cb-rpxgt" Sep 10 13:31:46.878683 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:31:46.878660 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2a5dff35-e18a-44cc-a1a9-ccdc505d818d-istio-envoy\") pod \"e2e-explicit-a26db0-openshift-default-74c74695cb-rpxgt\" (UID: \"2a5dff35-e18a-44cc-a1a9-ccdc505d818d\") " pod="openshift-ingress/e2e-explicit-a26db0-openshift-default-74c74695cb-rpxgt" Sep 10 13:31:46.879202 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:31:46.879131 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2a5dff35-e18a-44cc-a1a9-ccdc505d818d-istio-podinfo\") pod \"e2e-explicit-a26db0-openshift-default-74c74695cb-rpxgt\" (UID: \"2a5dff35-e18a-44cc-a1a9-ccdc505d818d\") " pod="openshift-ingress/e2e-explicit-a26db0-openshift-default-74c74695cb-rpxgt" Sep 10 13:31:46.892907 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:31:46.892855 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ppccs\" (UniqueName: \"kubernetes.io/projected/2a5dff35-e18a-44cc-a1a9-ccdc505d818d-kube-api-access-ppccs\") pod \"e2e-explicit-a26db0-openshift-default-74c74695cb-rpxgt\" (UID: \"2a5dff35-e18a-44cc-a1a9-ccdc505d818d\") " pod="openshift-ingress/e2e-explicit-a26db0-openshift-default-74c74695cb-rpxgt" Sep 10 13:31:46.893590 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:31:46.893571 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2a5dff35-e18a-44cc-a1a9-ccdc505d818d-istio-token\") pod \"e2e-explicit-a26db0-openshift-default-74c74695cb-rpxgt\" (UID: \"2a5dff35-e18a-44cc-a1a9-ccdc505d818d\") " pod="openshift-ingress/e2e-explicit-a26db0-openshift-default-74c74695cb-rpxgt" Sep 10 13:31:46.957164 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:31:46.957141 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-explicit-a26db0-openshift-default-74c74695cb-rpxgt" Sep 10 13:31:47.257941 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:31:47.257904 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-explicit-a26db0-openshift-default-74c74695cb-rpxgt"] Sep 10 13:31:47.259439 ip-10-0-139-174 kubenswrapper[2557]: W0910 13:31:47.259377 2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2a5dff35_e18a_44cc_a1a9_ccdc505d818d.slice/crio-fb9e71ba88455cac883b30394c04b4be61df408c4a4d5153bc5e62caaefc374a WatchSource:0}: Error finding container fb9e71ba88455cac883b30394c04b4be61df408c4a4d5153bc5e62caaefc374a: Status 404 returned error can't find the container with id fb9e71ba88455cac883b30394c04b4be61df408c4a4d5153bc5e62caaefc374a Sep 10 13:31:47.261729 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:31:47.261694 2557 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892152Ki","pods":"250"} Sep 10 13:31:47.261834 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:31:47.261759 2557 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892152Ki","pods":"250"} Sep 10 13:31:47.261834 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:31:47.261788 2557 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892152Ki","pods":"250"} Sep 10 13:31:48.032236 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:31:48.032200 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-explicit-a26db0-openshift-default-74c74695cb-rpxgt" event={"ID":"2a5dff35-e18a-44cc-a1a9-ccdc505d818d","Type":"ContainerStarted","Data":"78b77a3ee19dd39bfd8d6e1de845ddacfc6b4ef5fea454096692cecb5a71ed2d"} Sep 10 13:31:48.032632 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:31:48.032254 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-explicit-a26db0-openshift-default-74c74695cb-rpxgt" event={"ID":"2a5dff35-e18a-44cc-a1a9-ccdc505d818d","Type":"ContainerStarted","Data":"fb9e71ba88455cac883b30394c04b4be61df408c4a4d5153bc5e62caaefc374a"} Sep 10 13:31:48.072733 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:31:48.072676 2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-explicit-a26db0-openshift-default-74c74695cb-rpxgt" podStartSLOduration=2.072657803 podStartE2EDuration="2.072657803s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 13:31:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 13:31:48.069552289 +0000 UTC m=+868.376546706" watchObservedRunningTime="2026-09-10 13:31:48.072657803 +0000 UTC m=+868.379652217" Sep 10 13:31:48.957439 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:31:48.957384 2557 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-explicit-a26db0-openshift-default-74c74695cb-rpxgt" Sep 10 13:31:48.962636 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:31:48.962607 2557 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-explicit-a26db0-openshift-default-74c74695cb-rpxgt" Sep 10 13:31:49.043159 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:31:49.042972 2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-explicit-a26db0-openshift-default-74c74695cb-rpxgt" Sep 10 13:31:49.045460 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:31:49.045382 2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-explicit-a26db0-openshift-default-74c74695cb-rpxgt" Sep 10 13:31:51.000376 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:31:51.000349 2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-autores-af59ca-openshift-default-887c6d76d-t4qb9" Sep 10 13:31:51.022190 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:31:51.022129 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/22c3a6a3-69e4-43db-8768-525668cb1928-istio-podinfo\") pod \"22c3a6a3-69e4-43db-8768-525668cb1928\" (UID: \"22c3a6a3-69e4-43db-8768-525668cb1928\") " Sep 10 13:31:51.022190 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:31:51.022179 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/22c3a6a3-69e4-43db-8768-525668cb1928-workload-certs\") pod \"22c3a6a3-69e4-43db-8768-525668cb1928\" (UID: \"22c3a6a3-69e4-43db-8768-525668cb1928\") " Sep 10 13:31:51.022352 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:31:51.022222 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/22c3a6a3-69e4-43db-8768-525668cb1928-istio-token\") pod \"22c3a6a3-69e4-43db-8768-525668cb1928\" (UID: \"22c3a6a3-69e4-43db-8768-525668cb1928\") " Sep 10 13:31:51.022352 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:31:51.022248 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/22c3a6a3-69e4-43db-8768-525668cb1928-workload-socket\") pod \"22c3a6a3-69e4-43db-8768-525668cb1928\" (UID: \"22c3a6a3-69e4-43db-8768-525668cb1928\") " Sep 10 13:31:51.022352 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:31:51.022273 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/22c3a6a3-69e4-43db-8768-525668cb1928-credential-socket\") pod \"22c3a6a3-69e4-43db-8768-525668cb1928\" (UID: \"22c3a6a3-69e4-43db-8768-525668cb1928\") " Sep 10 13:31:51.022352 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:31:51.022336 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wkd9n\" (UniqueName: \"kubernetes.io/projected/22c3a6a3-69e4-43db-8768-525668cb1928-kube-api-access-wkd9n\") pod \"22c3a6a3-69e4-43db-8768-525668cb1928\" (UID: \"22c3a6a3-69e4-43db-8768-525668cb1928\") " Sep 10 13:31:51.022585 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:31:51.022367 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/22c3a6a3-69e4-43db-8768-525668cb1928-istiod-ca-cert\") pod \"22c3a6a3-69e4-43db-8768-525668cb1928\" (UID: \"22c3a6a3-69e4-43db-8768-525668cb1928\") " Sep 10 13:31:51.022585 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:31:51.022395 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/22c3a6a3-69e4-43db-8768-525668cb1928-istio-data\") pod \"22c3a6a3-69e4-43db-8768-525668cb1928\" (UID: \"22c3a6a3-69e4-43db-8768-525668cb1928\") " Sep 10 13:31:51.022585 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:31:51.022439 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/22c3a6a3-69e4-43db-8768-525668cb1928-istio-envoy\") pod \"22c3a6a3-69e4-43db-8768-525668cb1928\" (UID: \"22c3a6a3-69e4-43db-8768-525668cb1928\") " Sep 10 13:31:51.022585 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:31:51.022559 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/22c3a6a3-69e4-43db-8768-525668cb1928-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "22c3a6a3-69e4-43db-8768-525668cb1928" (UID: "22c3a6a3-69e4-43db-8768-525668cb1928"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 13:31:51.022793 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:31:51.022566 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/22c3a6a3-69e4-43db-8768-525668cb1928-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "22c3a6a3-69e4-43db-8768-525668cb1928" (UID: "22c3a6a3-69e4-43db-8768-525668cb1928"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 13:31:51.022863 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:31:51.022837 2557 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/22c3a6a3-69e4-43db-8768-525668cb1928-workload-certs\") on node \"ip-10-0-139-174.ec2.internal\" DevicePath \"\"" Sep 10 13:31:51.022927 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:31:51.022874 2557 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/22c3a6a3-69e4-43db-8768-525668cb1928-workload-socket\") on node \"ip-10-0-139-174.ec2.internal\" DevicePath \"\"" Sep 10 13:31:51.022927 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:31:51.022836 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/22c3a6a3-69e4-43db-8768-525668cb1928-istio-data" (OuterVolumeSpecName: "istio-data") pod "22c3a6a3-69e4-43db-8768-525668cb1928" (UID: "22c3a6a3-69e4-43db-8768-525668cb1928"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 13:31:51.022927 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:31:51.022869 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/22c3a6a3-69e4-43db-8768-525668cb1928-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "22c3a6a3-69e4-43db-8768-525668cb1928" (UID: "22c3a6a3-69e4-43db-8768-525668cb1928"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 10 13:31:51.023083 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:31:51.022693 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/22c3a6a3-69e4-43db-8768-525668cb1928-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "22c3a6a3-69e4-43db-8768-525668cb1928" (UID: "22c3a6a3-69e4-43db-8768-525668cb1928"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 13:31:51.024925 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:31:51.024866 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/22c3a6a3-69e4-43db-8768-525668cb1928-istio-token" (OuterVolumeSpecName: "istio-token") pod "22c3a6a3-69e4-43db-8768-525668cb1928" (UID: "22c3a6a3-69e4-43db-8768-525668cb1928"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 10 13:31:51.025566 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:31:51.025530 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/22c3a6a3-69e4-43db-8768-525668cb1928-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "22c3a6a3-69e4-43db-8768-525668cb1928" (UID: "22c3a6a3-69e4-43db-8768-525668cb1928"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Sep 10 13:31:51.026167 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:31:51.026142 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/22c3a6a3-69e4-43db-8768-525668cb1928-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "22c3a6a3-69e4-43db-8768-525668cb1928" (UID: "22c3a6a3-69e4-43db-8768-525668cb1928"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 13:31:51.026277 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:31:51.026189 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/22c3a6a3-69e4-43db-8768-525668cb1928-kube-api-access-wkd9n" (OuterVolumeSpecName: "kube-api-access-wkd9n") pod "22c3a6a3-69e4-43db-8768-525668cb1928" (UID: "22c3a6a3-69e4-43db-8768-525668cb1928"). InnerVolumeSpecName "kube-api-access-wkd9n". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 10 13:31:51.058022 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:31:51.057994 2557 generic.go:356] "Generic (PLEG): container finished" podID="22c3a6a3-69e4-43db-8768-525668cb1928" containerID="caffa4bde3e2aecd434145033c8a8291c60306c41704f29f98bed8fb7fea89bb" exitCode=0 Sep 10 13:31:51.058138 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:31:51.058039 2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-autores-af59ca-openshift-default-887c6d76d-t4qb9" Sep 10 13:31:51.058138 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:31:51.058092 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-autores-af59ca-openshift-default-887c6d76d-t4qb9" event={"ID":"22c3a6a3-69e4-43db-8768-525668cb1928","Type":"ContainerDied","Data":"caffa4bde3e2aecd434145033c8a8291c60306c41704f29f98bed8fb7fea89bb"} Sep 10 13:31:51.058228 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:31:51.058190 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-autores-af59ca-openshift-default-887c6d76d-t4qb9" event={"ID":"22c3a6a3-69e4-43db-8768-525668cb1928","Type":"ContainerDied","Data":"016c34660714d2f5ac9f05e86ca451278e040d73451a86f6838c25f61deaf50a"} Sep 10 13:31:51.058284 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:31:51.058227 2557 scope.go:117] "RemoveContainer" containerID="caffa4bde3e2aecd434145033c8a8291c60306c41704f29f98bed8fb7fea89bb" Sep 10 13:31:51.068505 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:31:51.068487 2557 scope.go:117] "RemoveContainer" containerID="1135f5e5d87cb02f53d9744e567fffc5f706cae4f2503f2886bb9597b97f0349" Sep 10 13:31:51.079552 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:31:51.079536 2557 scope.go:117] "RemoveContainer" containerID="caffa4bde3e2aecd434145033c8a8291c60306c41704f29f98bed8fb7fea89bb" Sep 10 13:31:51.080037 ip-10-0-139-174 kubenswrapper[2557]: E0910 13:31:51.080014 2557 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"caffa4bde3e2aecd434145033c8a8291c60306c41704f29f98bed8fb7fea89bb\": container with ID starting with caffa4bde3e2aecd434145033c8a8291c60306c41704f29f98bed8fb7fea89bb not found: ID does not exist" containerID="caffa4bde3e2aecd434145033c8a8291c60306c41704f29f98bed8fb7fea89bb" Sep 10 13:31:51.080099 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:31:51.080046 2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"caffa4bde3e2aecd434145033c8a8291c60306c41704f29f98bed8fb7fea89bb"} err="failed to get container status \"caffa4bde3e2aecd434145033c8a8291c60306c41704f29f98bed8fb7fea89bb\": rpc error: code = NotFound desc = could not find container \"caffa4bde3e2aecd434145033c8a8291c60306c41704f29f98bed8fb7fea89bb\": container with ID starting with caffa4bde3e2aecd434145033c8a8291c60306c41704f29f98bed8fb7fea89bb not found: ID does not exist" Sep 10 13:31:51.080099 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:31:51.080064 2557 scope.go:117] "RemoveContainer" containerID="1135f5e5d87cb02f53d9744e567fffc5f706cae4f2503f2886bb9597b97f0349" Sep 10 13:31:51.080372 ip-10-0-139-174 kubenswrapper[2557]: E0910 13:31:51.080354 2557 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1135f5e5d87cb02f53d9744e567fffc5f706cae4f2503f2886bb9597b97f0349\": container with ID starting with 1135f5e5d87cb02f53d9744e567fffc5f706cae4f2503f2886bb9597b97f0349 not found: ID does not exist" containerID="1135f5e5d87cb02f53d9744e567fffc5f706cae4f2503f2886bb9597b97f0349" Sep 10 13:31:51.080453 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:31:51.080385 2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1135f5e5d87cb02f53d9744e567fffc5f706cae4f2503f2886bb9597b97f0349"} err="failed to get container status \"1135f5e5d87cb02f53d9744e567fffc5f706cae4f2503f2886bb9597b97f0349\": rpc error: code = NotFound desc = could not find container \"1135f5e5d87cb02f53d9744e567fffc5f706cae4f2503f2886bb9597b97f0349\": container with ID starting with 1135f5e5d87cb02f53d9744e567fffc5f706cae4f2503f2886bb9597b97f0349 not found: ID does not exist" Sep 10 13:31:51.084721 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:31:51.084700 2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-autores-af59ca-openshift-default-887c6d76d-t4qb9"] Sep 10 13:31:51.087221 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:31:51.087201 2557 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-autores-af59ca-openshift-default-887c6d76d-t4qb9"] Sep 10 13:31:51.123383 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:31:51.123354 2557 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/22c3a6a3-69e4-43db-8768-525668cb1928-istio-envoy\") on node \"ip-10-0-139-174.ec2.internal\" DevicePath \"\"" Sep 10 13:31:51.123383 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:31:51.123378 2557 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/22c3a6a3-69e4-43db-8768-525668cb1928-istio-podinfo\") on node \"ip-10-0-139-174.ec2.internal\" DevicePath \"\"" Sep 10 13:31:51.123383 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:31:51.123386 2557 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/22c3a6a3-69e4-43db-8768-525668cb1928-istio-token\") on node \"ip-10-0-139-174.ec2.internal\" DevicePath \"\"" Sep 10 13:31:51.123598 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:31:51.123395 2557 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/22c3a6a3-69e4-43db-8768-525668cb1928-credential-socket\") on node \"ip-10-0-139-174.ec2.internal\" DevicePath \"\"" Sep 10 13:31:51.123598 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:31:51.123405 2557 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wkd9n\" (UniqueName: \"kubernetes.io/projected/22c3a6a3-69e4-43db-8768-525668cb1928-kube-api-access-wkd9n\") on node \"ip-10-0-139-174.ec2.internal\" DevicePath \"\"" Sep 10 13:31:51.123598 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:31:51.123444 2557 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/22c3a6a3-69e4-43db-8768-525668cb1928-istiod-ca-cert\") on node \"ip-10-0-139-174.ec2.internal\" DevicePath \"\"" Sep 10 13:31:51.123598 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:31:51.123458 2557 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/22c3a6a3-69e4-43db-8768-525668cb1928-istio-data\") on node \"ip-10-0-139-174.ec2.internal\" DevicePath \"\"" Sep 10 13:31:52.220942 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:31:52.220902 2557 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="22c3a6a3-69e4-43db-8768-525668cb1928" path="/var/lib/kubelet/pods/22c3a6a3-69e4-43db-8768-525668cb1928/volumes" Sep 10 13:32:12.212286 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:32:12.212246 2557 generic.go:356] "Generic (PLEG): container finished" podID="2a5dff35-e18a-44cc-a1a9-ccdc505d818d" containerID="78b77a3ee19dd39bfd8d6e1de845ddacfc6b4ef5fea454096692cecb5a71ed2d" exitCode=0 Sep 10 13:32:12.212767 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:32:12.212329 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-explicit-a26db0-openshift-default-74c74695cb-rpxgt" event={"ID":"2a5dff35-e18a-44cc-a1a9-ccdc505d818d","Type":"ContainerDied","Data":"78b77a3ee19dd39bfd8d6e1de845ddacfc6b4ef5fea454096692cecb5a71ed2d"} Sep 10 13:32:12.212828 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:32:12.212770 2557 scope.go:117] "RemoveContainer" containerID="78b77a3ee19dd39bfd8d6e1de845ddacfc6b4ef5fea454096692cecb5a71ed2d" Sep 10 13:32:12.213900 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:32:12.213871 2557 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892152Ki","pods":"250"} Sep 10 13:32:12.213997 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:32:12.213956 2557 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892152Ki","pods":"250"} Sep 10 13:32:12.214061 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:32:12.214007 2557 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892152Ki","pods":"250"} Sep 10 13:32:13.220054 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:32:13.220012 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-explicit-a26db0-openshift-default-74c74695cb-rpxgt" event={"ID":"2a5dff35-e18a-44cc-a1a9-ccdc505d818d","Type":"ContainerStarted","Data":"cf0a540dab4bff9918d50724eede8cc49a8d6371b6b924525cdda5cfe50862d2"} Sep 10 13:32:13.958668 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:32:13.958626 2557 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-explicit-a26db0-openshift-default-74c74695cb-rpxgt" Sep 10 13:32:13.958907 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:32:13.958762 2557 patch_prober.go:28] interesting pod/e2e-explicit-a26db0-openshift-default-74c74695cb-rpxgt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.59:15021/healthz/ready\": dial tcp 10.134.0.59:15021: connect: connection refused" start-of-body= Sep 10 13:32:13.958907 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:32:13.958820 2557 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-explicit-a26db0-openshift-default-74c74695cb-rpxgt" podUID="2a5dff35-e18a-44cc-a1a9-ccdc505d818d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.59:15021/healthz/ready\": dial tcp 10.134.0.59:15021: connect: connection refused" Sep 10 13:32:14.958371 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:32:14.958326 2557 patch_prober.go:28] interesting pod/e2e-explicit-a26db0-openshift-default-74c74695cb-rpxgt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.59:15021/healthz/ready\": dial tcp 10.134.0.59:15021: connect: connection refused" start-of-body= Sep 10 13:32:14.958881 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:32:14.958434 2557 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-explicit-a26db0-openshift-default-74c74695cb-rpxgt" podUID="2a5dff35-e18a-44cc-a1a9-ccdc505d818d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.59:15021/healthz/ready\": dial tcp 10.134.0.59:15021: connect: connection refused" Sep 10 13:32:15.958770 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:32:15.958252 2557 patch_prober.go:28] interesting pod/e2e-explicit-a26db0-openshift-default-74c74695cb-rpxgt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.59:15021/healthz/ready\": dial tcp 10.134.0.59:15021: connect: connection refused" start-of-body= Sep 10 13:32:15.958770 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:32:15.958336 2557 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-explicit-a26db0-openshift-default-74c74695cb-rpxgt" podUID="2a5dff35-e18a-44cc-a1a9-ccdc505d818d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.59:15021/healthz/ready\": dial tcp 10.134.0.59:15021: connect: connection refused" Sep 10 13:32:16.958599 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:32:16.958555 2557 patch_prober.go:28] interesting pod/e2e-explicit-a26db0-openshift-default-74c74695cb-rpxgt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.59:15021/healthz/ready\": dial tcp 10.134.0.59:15021: connect: connection refused" start-of-body= Sep 10 13:32:16.958819 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:32:16.958636 2557 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-explicit-a26db0-openshift-default-74c74695cb-rpxgt" podUID="2a5dff35-e18a-44cc-a1a9-ccdc505d818d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.59:15021/healthz/ready\": dial tcp 10.134.0.59:15021: connect: connection refused" Sep 10 13:32:17.958283 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:32:17.958236 2557 patch_prober.go:28] interesting pod/e2e-explicit-a26db0-openshift-default-74c74695cb-rpxgt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.59:15021/healthz/ready\": dial tcp 10.134.0.59:15021: connect: connection refused" start-of-body= Sep 10 13:32:17.958530 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:32:17.958305 2557 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-explicit-a26db0-openshift-default-74c74695cb-rpxgt" podUID="2a5dff35-e18a-44cc-a1a9-ccdc505d818d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.59:15021/healthz/ready\": dial tcp 10.134.0.59:15021: connect: connection refused" Sep 10 13:32:18.962287 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:32:18.962258 2557 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-explicit-a26db0-openshift-default-74c74695cb-rpxgt" Sep 10 13:32:18.962676 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:32:18.962529 2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-explicit-a26db0-openshift-default-74c74695cb-rpxgt" Sep 10 13:32:18.963157 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:32:18.963138 2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-explicit-a26db0-openshift-default-74c74695cb-rpxgt" Sep 10 13:32:20.184778 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:32:20.184753 2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-92n4q_720acda6-0b15-49ac-ab4e-60de70c4324a/ovn-acl-logging/0.log" Sep 10 13:32:20.187662 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:32:20.187637 2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-92n4q_720acda6-0b15-49ac-ab4e-60de70c4324a/ovn-acl-logging/0.log" Sep 10 13:33:03.910075 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:33:03.910033 2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-a26db0-dvh5c"] Sep 10 13:33:03.910678 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:33:03.910653 2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="22c3a6a3-69e4-43db-8768-525668cb1928" containerName="istio-proxy" Sep 10 13:33:03.910678 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:33:03.910673 2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="22c3a6a3-69e4-43db-8768-525668cb1928" containerName="istio-proxy" Sep 10 13:33:03.910808 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:33:03.910690 2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="22c3a6a3-69e4-43db-8768-525668cb1928" containerName="istio-proxy" Sep 10 13:33:03.910808 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:33:03.910698 2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="22c3a6a3-69e4-43db-8768-525668cb1928" containerName="istio-proxy" Sep 10 13:33:03.910909 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:33:03.910817 2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="27bdc7ae-a8d0-43fb-8d70-7a247ef9db39" containerName="istio-proxy" Sep 10 13:33:03.910909 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:33:03.910833 2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="22c3a6a3-69e4-43db-8768-525668cb1928" containerName="istio-proxy" Sep 10 13:33:03.912934 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:33:03.912911 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-a26db0-dvh5c" Sep 10 13:33:03.916212 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:33:03.916190 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-wtblv\"" Sep 10 13:33:03.916759 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:33:03.916743 2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Sep 10 13:33:03.927071 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:33:03.927051 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-a26db0-dvh5c"] Sep 10 13:33:04.003630 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:33:04.003601 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0cc49e4b-3880-4e15-92a9-79349ec7e379-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-explicit-a26db0-dvh5c\" (UID: \"0cc49e4b-3880-4e15-92a9-79349ec7e379\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-a26db0-dvh5c" Sep 10 13:33:04.104489 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:33:04.104458 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0cc49e4b-3880-4e15-92a9-79349ec7e379-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-explicit-a26db0-dvh5c\" (UID: \"0cc49e4b-3880-4e15-92a9-79349ec7e379\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-a26db0-dvh5c" Sep 10 13:33:04.105052 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:33:04.105034 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0cc49e4b-3880-4e15-92a9-79349ec7e379-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-explicit-a26db0-dvh5c\" (UID: \"0cc49e4b-3880-4e15-92a9-79349ec7e379\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-a26db0-dvh5c" Sep 10 13:33:04.222520 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:33:04.222499 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-a26db0-dvh5c" Sep 10 13:33:04.526996 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:33:04.526972 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-a26db0-dvh5c"] Sep 10 13:33:04.529017 ip-10-0-139-174 kubenswrapper[2557]: W0910 13:33:04.528990 2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0cc49e4b_3880_4e15_92a9_79349ec7e379.slice/crio-c677c132dd9cdd0d3431d011cfda4ef0b50245b88b7f50ab7ef1cff730e3efd2 WatchSource:0}: Error finding container c677c132dd9cdd0d3431d011cfda4ef0b50245b88b7f50ab7ef1cff730e3efd2: Status 404 returned error can't find the container with id c677c132dd9cdd0d3431d011cfda4ef0b50245b88b7f50ab7ef1cff730e3efd2 Sep 10 13:33:05.527801 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:33:05.527710 2557 generic.go:356] "Generic (PLEG): container finished" podID="0cc49e4b-3880-4e15-92a9-79349ec7e379" containerID="3c415ef089657725f6990a29870fcb11203788ca88fa07413cfc4ecdfc339cee" exitCode=0 Sep 10 13:33:05.527801 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:33:05.527767 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-a26db0-dvh5c" event={"ID":"0cc49e4b-3880-4e15-92a9-79349ec7e379","Type":"ContainerDied","Data":"3c415ef089657725f6990a29870fcb11203788ca88fa07413cfc4ecdfc339cee"} Sep 10 13:33:05.528183 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:33:05.527819 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-a26db0-dvh5c" event={"ID":"0cc49e4b-3880-4e15-92a9-79349ec7e379","Type":"ContainerStarted","Data":"c677c132dd9cdd0d3431d011cfda4ef0b50245b88b7f50ab7ef1cff730e3efd2"} Sep 10 13:33:06.666641 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:33:06.666612 2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-a26db0-dvh5c" Sep 10 13:33:06.828461 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:33:06.828368 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0cc49e4b-3880-4e15-92a9-79349ec7e379-maas-api-service-ca\") pod \"0cc49e4b-3880-4e15-92a9-79349ec7e379\" (UID: \"0cc49e4b-3880-4e15-92a9-79349ec7e379\") " Sep 10 13:33:06.828690 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:33:06.828669 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0cc49e4b-3880-4e15-92a9-79349ec7e379-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "0cc49e4b-3880-4e15-92a9-79349ec7e379" (UID: "0cc49e4b-3880-4e15-92a9-79349ec7e379"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 10 13:33:06.930036 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:33:06.930009 2557 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0cc49e4b-3880-4e15-92a9-79349ec7e379-maas-api-service-ca\") on node \"ip-10-0-139-174.ec2.internal\" DevicePath \"\"" Sep 10 13:33:07.541888 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:33:07.541851 2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-a26db0-dvh5c" Sep 10 13:33:07.542090 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:33:07.541898 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-a26db0-dvh5c" event={"ID":"0cc49e4b-3880-4e15-92a9-79349ec7e379","Type":"ContainerDied","Data":"c677c132dd9cdd0d3431d011cfda4ef0b50245b88b7f50ab7ef1cff730e3efd2"} Sep 10 13:33:07.542090 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:33:07.541941 2557 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c677c132dd9cdd0d3431d011cfda4ef0b50245b88b7f50ab7ef1cff730e3efd2" Sep 10 13:33:15.566105 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:33:15.566065 2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-a26db0-dvh5c"] Sep 10 13:33:15.576992 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:33:15.576946 2557 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-a26db0-dvh5c"] Sep 10 13:33:16.219400 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:33:16.219359 2557 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0cc49e4b-3880-4e15-92a9-79349ec7e379" path="/var/lib/kubelet/pods/0cc49e4b-3880-4e15-92a9-79349ec7e379/volumes" Sep 10 13:33:24.130537 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:33:24.130493 2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-explicit-a26db0-openshift-default-74c74695cb-rpxgt"] Sep 10 13:33:24.130962 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:33:24.130713 2557 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-explicit-a26db0-openshift-default-74c74695cb-rpxgt" podUID="2a5dff35-e18a-44cc-a1a9-ccdc505d818d" containerName="istio-proxy" containerID="cri-o://cf0a540dab4bff9918d50724eede8cc49a8d6371b6b924525cdda5cfe50862d2" gracePeriod=30 Sep 10 13:33:29.381620 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:33:29.381595 2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-explicit-a26db0-openshift-default-74c74695cb-rpxgt" Sep 10 13:33:29.448503 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:33:29.448475 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2a5dff35-e18a-44cc-a1a9-ccdc505d818d-credential-socket\") pod \"2a5dff35-e18a-44cc-a1a9-ccdc505d818d\" (UID: \"2a5dff35-e18a-44cc-a1a9-ccdc505d818d\") " Sep 10 13:33:29.448658 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:33:29.448537 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2a5dff35-e18a-44cc-a1a9-ccdc505d818d-istiod-ca-cert\") pod \"2a5dff35-e18a-44cc-a1a9-ccdc505d818d\" (UID: \"2a5dff35-e18a-44cc-a1a9-ccdc505d818d\") " Sep 10 13:33:29.448658 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:33:29.448575 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ppccs\" (UniqueName: \"kubernetes.io/projected/2a5dff35-e18a-44cc-a1a9-ccdc505d818d-kube-api-access-ppccs\") pod \"2a5dff35-e18a-44cc-a1a9-ccdc505d818d\" (UID: \"2a5dff35-e18a-44cc-a1a9-ccdc505d818d\") " Sep 10 13:33:29.448658 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:33:29.448608 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2a5dff35-e18a-44cc-a1a9-ccdc505d818d-workload-socket\") pod \"2a5dff35-e18a-44cc-a1a9-ccdc505d818d\" (UID: \"2a5dff35-e18a-44cc-a1a9-ccdc505d818d\") " Sep 10 13:33:29.448658 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:33:29.448627 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2a5dff35-e18a-44cc-a1a9-ccdc505d818d-istio-podinfo\") pod \"2a5dff35-e18a-44cc-a1a9-ccdc505d818d\" (UID: \"2a5dff35-e18a-44cc-a1a9-ccdc505d818d\") " Sep 10 13:33:29.448658 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:33:29.448652 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2a5dff35-e18a-44cc-a1a9-ccdc505d818d-istio-envoy\") pod \"2a5dff35-e18a-44cc-a1a9-ccdc505d818d\" (UID: \"2a5dff35-e18a-44cc-a1a9-ccdc505d818d\") " Sep 10 13:33:29.448979 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:33:29.448687 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2a5dff35-e18a-44cc-a1a9-ccdc505d818d-istio-token\") pod \"2a5dff35-e18a-44cc-a1a9-ccdc505d818d\" (UID: \"2a5dff35-e18a-44cc-a1a9-ccdc505d818d\") " Sep 10 13:33:29.448979 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:33:29.448713 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2a5dff35-e18a-44cc-a1a9-ccdc505d818d-workload-certs\") pod \"2a5dff35-e18a-44cc-a1a9-ccdc505d818d\" (UID: \"2a5dff35-e18a-44cc-a1a9-ccdc505d818d\") " Sep 10 13:33:29.448979 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:33:29.448744 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2a5dff35-e18a-44cc-a1a9-ccdc505d818d-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "2a5dff35-e18a-44cc-a1a9-ccdc505d818d" (UID: "2a5dff35-e18a-44cc-a1a9-ccdc505d818d"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 13:33:29.448979 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:33:29.448770 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2a5dff35-e18a-44cc-a1a9-ccdc505d818d-istio-data\") pod \"2a5dff35-e18a-44cc-a1a9-ccdc505d818d\" (UID: \"2a5dff35-e18a-44cc-a1a9-ccdc505d818d\") " Sep 10 13:33:29.448979 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:33:29.448916 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2a5dff35-e18a-44cc-a1a9-ccdc505d818d-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "2a5dff35-e18a-44cc-a1a9-ccdc505d818d" (UID: "2a5dff35-e18a-44cc-a1a9-ccdc505d818d"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 13:33:29.449240 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:33:29.449102 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2a5dff35-e18a-44cc-a1a9-ccdc505d818d-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "2a5dff35-e18a-44cc-a1a9-ccdc505d818d" (UID: "2a5dff35-e18a-44cc-a1a9-ccdc505d818d"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 13:33:29.449240 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:33:29.449030 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2a5dff35-e18a-44cc-a1a9-ccdc505d818d-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "2a5dff35-e18a-44cc-a1a9-ccdc505d818d" (UID: "2a5dff35-e18a-44cc-a1a9-ccdc505d818d"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 10 13:33:29.449240 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:33:29.449149 2557 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2a5dff35-e18a-44cc-a1a9-ccdc505d818d-credential-socket\") on node \"ip-10-0-139-174.ec2.internal\" DevicePath \"\"" Sep 10 13:33:29.449240 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:33:29.449161 2557 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2a5dff35-e18a-44cc-a1a9-ccdc505d818d-workload-socket\") on node \"ip-10-0-139-174.ec2.internal\" DevicePath \"\"" Sep 10 13:33:29.449366 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:33:29.449233 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2a5dff35-e18a-44cc-a1a9-ccdc505d818d-istio-data" (OuterVolumeSpecName: "istio-data") pod "2a5dff35-e18a-44cc-a1a9-ccdc505d818d" (UID: "2a5dff35-e18a-44cc-a1a9-ccdc505d818d"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 13:33:29.451348 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:33:29.451317 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2a5dff35-e18a-44cc-a1a9-ccdc505d818d-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "2a5dff35-e18a-44cc-a1a9-ccdc505d818d" (UID: "2a5dff35-e18a-44cc-a1a9-ccdc505d818d"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 13:33:29.451490 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:33:29.451439 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/2a5dff35-e18a-44cc-a1a9-ccdc505d818d-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "2a5dff35-e18a-44cc-a1a9-ccdc505d818d" (UID: "2a5dff35-e18a-44cc-a1a9-ccdc505d818d"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Sep 10 13:33:29.451571 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:33:29.451551 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2a5dff35-e18a-44cc-a1a9-ccdc505d818d-kube-api-access-ppccs" (OuterVolumeSpecName: "kube-api-access-ppccs") pod "2a5dff35-e18a-44cc-a1a9-ccdc505d818d" (UID: "2a5dff35-e18a-44cc-a1a9-ccdc505d818d"). InnerVolumeSpecName "kube-api-access-ppccs". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 10 13:33:29.451666 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:33:29.451646 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2a5dff35-e18a-44cc-a1a9-ccdc505d818d-istio-token" (OuterVolumeSpecName: "istio-token") pod "2a5dff35-e18a-44cc-a1a9-ccdc505d818d" (UID: "2a5dff35-e18a-44cc-a1a9-ccdc505d818d"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 10 13:33:29.555873 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:33:29.551181 2557 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2a5dff35-e18a-44cc-a1a9-ccdc505d818d-istiod-ca-cert\") on node \"ip-10-0-139-174.ec2.internal\" DevicePath \"\"" Sep 10 13:33:29.555873 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:33:29.551227 2557 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ppccs\" (UniqueName: \"kubernetes.io/projected/2a5dff35-e18a-44cc-a1a9-ccdc505d818d-kube-api-access-ppccs\") on node \"ip-10-0-139-174.ec2.internal\" DevicePath \"\"" Sep 10 13:33:29.555873 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:33:29.551242 2557 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2a5dff35-e18a-44cc-a1a9-ccdc505d818d-istio-podinfo\") on node \"ip-10-0-139-174.ec2.internal\" DevicePath \"\"" Sep 10 13:33:29.555873 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:33:29.551256 2557 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2a5dff35-e18a-44cc-a1a9-ccdc505d818d-istio-envoy\") on node \"ip-10-0-139-174.ec2.internal\" DevicePath \"\"" Sep 10 13:33:29.555873 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:33:29.551273 2557 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2a5dff35-e18a-44cc-a1a9-ccdc505d818d-istio-token\") on node \"ip-10-0-139-174.ec2.internal\" DevicePath \"\"" Sep 10 13:33:29.555873 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:33:29.551290 2557 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2a5dff35-e18a-44cc-a1a9-ccdc505d818d-workload-certs\") on node \"ip-10-0-139-174.ec2.internal\" DevicePath \"\"" Sep 10 13:33:29.555873 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:33:29.551302 2557 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2a5dff35-e18a-44cc-a1a9-ccdc505d818d-istio-data\") on node \"ip-10-0-139-174.ec2.internal\" DevicePath \"\"" Sep 10 13:33:29.679902 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:33:29.679866 2557 generic.go:356] "Generic (PLEG): container finished" podID="2a5dff35-e18a-44cc-a1a9-ccdc505d818d" containerID="cf0a540dab4bff9918d50724eede8cc49a8d6371b6b924525cdda5cfe50862d2" exitCode=0 Sep 10 13:33:29.680067 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:33:29.679922 2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-explicit-a26db0-openshift-default-74c74695cb-rpxgt" Sep 10 13:33:29.680067 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:33:29.679954 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-explicit-a26db0-openshift-default-74c74695cb-rpxgt" event={"ID":"2a5dff35-e18a-44cc-a1a9-ccdc505d818d","Type":"ContainerDied","Data":"cf0a540dab4bff9918d50724eede8cc49a8d6371b6b924525cdda5cfe50862d2"} Sep 10 13:33:29.680067 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:33:29.680031 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-explicit-a26db0-openshift-default-74c74695cb-rpxgt" event={"ID":"2a5dff35-e18a-44cc-a1a9-ccdc505d818d","Type":"ContainerDied","Data":"fb9e71ba88455cac883b30394c04b4be61df408c4a4d5153bc5e62caaefc374a"} Sep 10 13:33:29.680067 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:33:29.680055 2557 scope.go:117] "RemoveContainer" containerID="cf0a540dab4bff9918d50724eede8cc49a8d6371b6b924525cdda5cfe50862d2" Sep 10 13:33:29.690340 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:33:29.690323 2557 scope.go:117] "RemoveContainer" containerID="78b77a3ee19dd39bfd8d6e1de845ddacfc6b4ef5fea454096692cecb5a71ed2d" Sep 10 13:33:29.701368 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:33:29.701341 2557 scope.go:117] "RemoveContainer" containerID="cf0a540dab4bff9918d50724eede8cc49a8d6371b6b924525cdda5cfe50862d2" Sep 10 13:33:29.701763 ip-10-0-139-174 kubenswrapper[2557]: E0910 13:33:29.701746 2557 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cf0a540dab4bff9918d50724eede8cc49a8d6371b6b924525cdda5cfe50862d2\": container with ID starting with cf0a540dab4bff9918d50724eede8cc49a8d6371b6b924525cdda5cfe50862d2 not found: ID does not exist" containerID="cf0a540dab4bff9918d50724eede8cc49a8d6371b6b924525cdda5cfe50862d2" Sep 10 13:33:29.701840 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:33:29.701772 2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cf0a540dab4bff9918d50724eede8cc49a8d6371b6b924525cdda5cfe50862d2"} err="failed to get container status \"cf0a540dab4bff9918d50724eede8cc49a8d6371b6b924525cdda5cfe50862d2\": rpc error: code = NotFound desc = could not find container \"cf0a540dab4bff9918d50724eede8cc49a8d6371b6b924525cdda5cfe50862d2\": container with ID starting with cf0a540dab4bff9918d50724eede8cc49a8d6371b6b924525cdda5cfe50862d2 not found: ID does not exist" Sep 10 13:33:29.701840 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:33:29.701790 2557 scope.go:117] "RemoveContainer" containerID="78b77a3ee19dd39bfd8d6e1de845ddacfc6b4ef5fea454096692cecb5a71ed2d" Sep 10 13:33:29.702126 ip-10-0-139-174 kubenswrapper[2557]: E0910 13:33:29.702098 2557 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"78b77a3ee19dd39bfd8d6e1de845ddacfc6b4ef5fea454096692cecb5a71ed2d\": container with ID starting with 78b77a3ee19dd39bfd8d6e1de845ddacfc6b4ef5fea454096692cecb5a71ed2d not found: ID does not exist" containerID="78b77a3ee19dd39bfd8d6e1de845ddacfc6b4ef5fea454096692cecb5a71ed2d" Sep 10 13:33:29.702183 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:33:29.702137 2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"78b77a3ee19dd39bfd8d6e1de845ddacfc6b4ef5fea454096692cecb5a71ed2d"} err="failed to get container status \"78b77a3ee19dd39bfd8d6e1de845ddacfc6b4ef5fea454096692cecb5a71ed2d\": rpc error: code = NotFound desc = could not find container \"78b77a3ee19dd39bfd8d6e1de845ddacfc6b4ef5fea454096692cecb5a71ed2d\": container with ID starting with 78b77a3ee19dd39bfd8d6e1de845ddacfc6b4ef5fea454096692cecb5a71ed2d not found: ID does not exist" Sep 10 13:33:29.710507 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:33:29.710477 2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-explicit-a26db0-openshift-default-74c74695cb-rpxgt"] Sep 10 13:33:29.745665 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:33:29.745623 2557 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-explicit-a26db0-openshift-default-74c74695cb-rpxgt"] Sep 10 13:33:30.220463 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:33:30.220411 2557 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2a5dff35-e18a-44cc-a1a9-ccdc505d818d" path="/var/lib/kubelet/pods/2a5dff35-e18a-44cc-a1a9-ccdc505d818d/volumes" Sep 10 13:33:58.103742 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:33:58.103548 2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-39df6e/test-model-39df6e-kserve-698d88cd99-xgkz7"] Sep 10 13:34:09.954685 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:34:09.954645 2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-39df6e-29817450-rhjv9"] Sep 10 13:34:09.970217 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:34:09.970189 2557 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-39df6e-29817450-rhjv9"] Sep 10 13:34:10.038597 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:34:10.038563 2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-39df6e-758db9c89f-d6wdl"] Sep 10 13:34:10.038790 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:34:10.038755 2557 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-a-39df6e-758db9c89f-d6wdl" podUID="516a2551-a59b-4435-8b6b-fdb75ca4d1d5" containerName="payload-processing" containerID="cri-o://7fa801e40c5a227a247bd78d5d448ba094bf0c1331f94dee6b65d6bf262d177e" gracePeriod=30 Sep 10 13:34:10.042163 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:34:10.042137 2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-39df6e-7bdcf56986-9czvg"] Sep 10 13:34:10.042548 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:34:10.042496 2557 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-39df6e-7bdcf56986-9czvg" podUID="ee2da889-3df3-4db7-abf0-9f5a75174e96" containerName="payload-pre-processing" containerID="cri-o://d531547914656f52029fd7af93bcbf2e6fb284576964283c0860da3bf39372d1" gracePeriod=30 Sep 10 13:34:10.195577 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:34:10.195527 2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-7f8babf4-openshift-default-74f9755b6d-dzdc9"] Sep 10 13:34:10.196299 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:34:10.196210 2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0cc49e4b-3880-4e15-92a9-79349ec7e379" containerName="revoke-keys" Sep 10 13:34:10.196299 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:34:10.196235 2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="0cc49e4b-3880-4e15-92a9-79349ec7e379" containerName="revoke-keys" Sep 10 13:34:10.196299 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:34:10.196257 2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2a5dff35-e18a-44cc-a1a9-ccdc505d818d" containerName="istio-proxy" Sep 10 13:34:10.196299 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:34:10.196268 2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="2a5dff35-e18a-44cc-a1a9-ccdc505d818d" containerName="istio-proxy" Sep 10 13:34:10.196299 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:34:10.196286 2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2a5dff35-e18a-44cc-a1a9-ccdc505d818d" containerName="istio-proxy" Sep 10 13:34:10.196299 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:34:10.196295 2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="2a5dff35-e18a-44cc-a1a9-ccdc505d818d" containerName="istio-proxy" Sep 10 13:34:10.196729 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:34:10.196441 2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="22c3a6a3-69e4-43db-8768-525668cb1928" containerName="istio-proxy" Sep 10 13:34:10.196729 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:34:10.196455 2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="2a5dff35-e18a-44cc-a1a9-ccdc505d818d" containerName="istio-proxy" Sep 10 13:34:10.196729 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:34:10.196463 2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="0cc49e4b-3880-4e15-92a9-79349ec7e379" containerName="revoke-keys" Sep 10 13:34:10.196729 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:34:10.196473 2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="2a5dff35-e18a-44cc-a1a9-ccdc505d818d" containerName="istio-proxy" Sep 10 13:34:10.200093 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:34:10.200070 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-7f8babf4-openshift-default-74f9755b6d-dzdc9" Sep 10 13:34:10.206081 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:34:10.205947 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-mt-7f8babf4-openshift-default-dockercfg-t9qk9\"" Sep 10 13:34:10.225557 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:34:10.225527 2557 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="36e73f75-14e4-4fe3-81cf-546e6a2c10e1" path="/var/lib/kubelet/pods/36e73f75-14e4-4fe3-81cf-546e6a2c10e1/volumes" Sep 10 13:34:10.234197 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:34:10.234154 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-7f8babf4-openshift-default-74f9755b6d-dzdc9"] Sep 10 13:34:10.302997 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:34:10.302966 2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-39df6e-758db9c89f-d6wdl" Sep 10 13:34:10.328456 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:34:10.328407 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/287603f3-efd3-46a9-bbfe-312f96974f76-credential-socket\") pod \"e2e-mt-7f8babf4-openshift-default-74f9755b6d-dzdc9\" (UID: \"287603f3-efd3-46a9-bbfe-312f96974f76\") " pod="openshift-ingress/e2e-mt-7f8babf4-openshift-default-74f9755b6d-dzdc9" Sep 10 13:34:10.328652 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:34:10.328633 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/287603f3-efd3-46a9-bbfe-312f96974f76-istio-envoy\") pod \"e2e-mt-7f8babf4-openshift-default-74f9755b6d-dzdc9\" (UID: \"287603f3-efd3-46a9-bbfe-312f96974f76\") " pod="openshift-ingress/e2e-mt-7f8babf4-openshift-default-74f9755b6d-dzdc9" Sep 10 13:34:10.328803 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:34:10.328782 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/287603f3-efd3-46a9-bbfe-312f96974f76-istio-podinfo\") pod \"e2e-mt-7f8babf4-openshift-default-74f9755b6d-dzdc9\" (UID: \"287603f3-efd3-46a9-bbfe-312f96974f76\") " pod="openshift-ingress/e2e-mt-7f8babf4-openshift-default-74f9755b6d-dzdc9" Sep 10 13:34:10.328933 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:34:10.328825 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/287603f3-efd3-46a9-bbfe-312f96974f76-istio-data\") pod \"e2e-mt-7f8babf4-openshift-default-74f9755b6d-dzdc9\" (UID: \"287603f3-efd3-46a9-bbfe-312f96974f76\") " pod="openshift-ingress/e2e-mt-7f8babf4-openshift-default-74f9755b6d-dzdc9" Sep 10 13:34:10.328933 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:34:10.328898 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/287603f3-efd3-46a9-bbfe-312f96974f76-istio-token\") pod \"e2e-mt-7f8babf4-openshift-default-74f9755b6d-dzdc9\" (UID: \"287603f3-efd3-46a9-bbfe-312f96974f76\") " pod="openshift-ingress/e2e-mt-7f8babf4-openshift-default-74f9755b6d-dzdc9" Sep 10 13:34:10.329098 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:34:10.328955 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/287603f3-efd3-46a9-bbfe-312f96974f76-workload-socket\") pod \"e2e-mt-7f8babf4-openshift-default-74f9755b6d-dzdc9\" (UID: \"287603f3-efd3-46a9-bbfe-312f96974f76\") " pod="openshift-ingress/e2e-mt-7f8babf4-openshift-default-74f9755b6d-dzdc9" Sep 10 13:34:10.329098 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:34:10.329058 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qn5rh\" (UniqueName: \"kubernetes.io/projected/287603f3-efd3-46a9-bbfe-312f96974f76-kube-api-access-qn5rh\") pod \"e2e-mt-7f8babf4-openshift-default-74f9755b6d-dzdc9\" (UID: \"287603f3-efd3-46a9-bbfe-312f96974f76\") " pod="openshift-ingress/e2e-mt-7f8babf4-openshift-default-74f9755b6d-dzdc9" Sep 10 13:34:10.329185 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:34:10.329108 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/287603f3-efd3-46a9-bbfe-312f96974f76-istiod-ca-cert\") pod \"e2e-mt-7f8babf4-openshift-default-74f9755b6d-dzdc9\" (UID: \"287603f3-efd3-46a9-bbfe-312f96974f76\") " pod="openshift-ingress/e2e-mt-7f8babf4-openshift-default-74f9755b6d-dzdc9" Sep 10 13:34:10.329230 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:34:10.329214 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/287603f3-efd3-46a9-bbfe-312f96974f76-workload-certs\") pod \"e2e-mt-7f8babf4-openshift-default-74f9755b6d-dzdc9\" (UID: \"287603f3-efd3-46a9-bbfe-312f96974f76\") " pod="openshift-ingress/e2e-mt-7f8babf4-openshift-default-74f9755b6d-dzdc9" Sep 10 13:34:10.335493 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:34:10.335469 2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-39df6e-7bdcf56986-9czvg" Sep 10 13:34:10.429606 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:34:10.429576 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ee2da889-3df3-4db7-abf0-9f5a75174e96-plugins-config-volume\") pod \"ee2da889-3df3-4db7-abf0-9f5a75174e96\" (UID: \"ee2da889-3df3-4db7-abf0-9f5a75174e96\") " Sep 10 13:34:10.429787 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:34:10.429658 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-zf8xl\" (UniqueName: \"kubernetes.io/projected/516a2551-a59b-4435-8b6b-fdb75ca4d1d5-kube-api-access-zf8xl\") pod \"516a2551-a59b-4435-8b6b-fdb75ca4d1d5\" (UID: \"516a2551-a59b-4435-8b6b-fdb75ca4d1d5\") " Sep 10 13:34:10.429787 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:34:10.429714 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5djkx\" (UniqueName: \"kubernetes.io/projected/ee2da889-3df3-4db7-abf0-9f5a75174e96-kube-api-access-5djkx\") pod \"ee2da889-3df3-4db7-abf0-9f5a75174e96\" (UID: \"ee2da889-3df3-4db7-abf0-9f5a75174e96\") " Sep 10 13:34:10.429916 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:34:10.429810 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/516a2551-a59b-4435-8b6b-fdb75ca4d1d5-plugins-config-volume\") pod \"516a2551-a59b-4435-8b6b-fdb75ca4d1d5\" (UID: \"516a2551-a59b-4435-8b6b-fdb75ca4d1d5\") " Sep 10 13:34:10.429970 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:34:10.429921 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/287603f3-efd3-46a9-bbfe-312f96974f76-workload-socket\") pod \"e2e-mt-7f8babf4-openshift-default-74f9755b6d-dzdc9\" (UID: \"287603f3-efd3-46a9-bbfe-312f96974f76\") " pod="openshift-ingress/e2e-mt-7f8babf4-openshift-default-74f9755b6d-dzdc9" Sep 10 13:34:10.430025 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:34:10.429969 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ee2da889-3df3-4db7-abf0-9f5a75174e96-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "ee2da889-3df3-4db7-abf0-9f5a75174e96" (UID: "ee2da889-3df3-4db7-abf0-9f5a75174e96"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 10 13:34:10.430025 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:34:10.429988 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qn5rh\" (UniqueName: \"kubernetes.io/projected/287603f3-efd3-46a9-bbfe-312f96974f76-kube-api-access-qn5rh\") pod \"e2e-mt-7f8babf4-openshift-default-74f9755b6d-dzdc9\" (UID: \"287603f3-efd3-46a9-bbfe-312f96974f76\") " pod="openshift-ingress/e2e-mt-7f8babf4-openshift-default-74f9755b6d-dzdc9" Sep 10 13:34:10.430132 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:34:10.430021 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/287603f3-efd3-46a9-bbfe-312f96974f76-istiod-ca-cert\") pod \"e2e-mt-7f8babf4-openshift-default-74f9755b6d-dzdc9\" (UID: \"287603f3-efd3-46a9-bbfe-312f96974f76\") " pod="openshift-ingress/e2e-mt-7f8babf4-openshift-default-74f9755b6d-dzdc9" Sep 10 13:34:10.430132 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:34:10.430124 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/287603f3-efd3-46a9-bbfe-312f96974f76-workload-certs\") pod \"e2e-mt-7f8babf4-openshift-default-74f9755b6d-dzdc9\" (UID: \"287603f3-efd3-46a9-bbfe-312f96974f76\") " pod="openshift-ingress/e2e-mt-7f8babf4-openshift-default-74f9755b6d-dzdc9" Sep 10 13:34:10.430240 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:34:10.430160 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/287603f3-efd3-46a9-bbfe-312f96974f76-credential-socket\") pod \"e2e-mt-7f8babf4-openshift-default-74f9755b6d-dzdc9\" (UID: \"287603f3-efd3-46a9-bbfe-312f96974f76\") " pod="openshift-ingress/e2e-mt-7f8babf4-openshift-default-74f9755b6d-dzdc9" Sep 10 13:34:10.430240 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:34:10.430191 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/287603f3-efd3-46a9-bbfe-312f96974f76-istio-envoy\") pod \"e2e-mt-7f8babf4-openshift-default-74f9755b6d-dzdc9\" (UID: \"287603f3-efd3-46a9-bbfe-312f96974f76\") " pod="openshift-ingress/e2e-mt-7f8babf4-openshift-default-74f9755b6d-dzdc9" Sep 10 13:34:10.430240 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:34:10.430221 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/516a2551-a59b-4435-8b6b-fdb75ca4d1d5-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "516a2551-a59b-4435-8b6b-fdb75ca4d1d5" (UID: "516a2551-a59b-4435-8b6b-fdb75ca4d1d5"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 10 13:34:10.430397 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:34:10.430240 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/287603f3-efd3-46a9-bbfe-312f96974f76-istio-podinfo\") pod \"e2e-mt-7f8babf4-openshift-default-74f9755b6d-dzdc9\" (UID: \"287603f3-efd3-46a9-bbfe-312f96974f76\") " pod="openshift-ingress/e2e-mt-7f8babf4-openshift-default-74f9755b6d-dzdc9" Sep 10 13:34:10.430397 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:34:10.430275 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/287603f3-efd3-46a9-bbfe-312f96974f76-istio-data\") pod \"e2e-mt-7f8babf4-openshift-default-74f9755b6d-dzdc9\" (UID: \"287603f3-efd3-46a9-bbfe-312f96974f76\") " pod="openshift-ingress/e2e-mt-7f8babf4-openshift-default-74f9755b6d-dzdc9" Sep 10 13:34:10.430397 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:34:10.430316 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/287603f3-efd3-46a9-bbfe-312f96974f76-workload-socket\") pod \"e2e-mt-7f8babf4-openshift-default-74f9755b6d-dzdc9\" (UID: \"287603f3-efd3-46a9-bbfe-312f96974f76\") " pod="openshift-ingress/e2e-mt-7f8babf4-openshift-default-74f9755b6d-dzdc9" Sep 10 13:34:10.430397 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:34:10.430320 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/287603f3-efd3-46a9-bbfe-312f96974f76-istio-token\") pod \"e2e-mt-7f8babf4-openshift-default-74f9755b6d-dzdc9\" (UID: \"287603f3-efd3-46a9-bbfe-312f96974f76\") " pod="openshift-ingress/e2e-mt-7f8babf4-openshift-default-74f9755b6d-dzdc9" Sep 10 13:34:10.430639 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:34:10.430449 2557 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/516a2551-a59b-4435-8b6b-fdb75ca4d1d5-plugins-config-volume\") on node \"ip-10-0-139-174.ec2.internal\" DevicePath \"\"" Sep 10 13:34:10.430639 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:34:10.430470 2557 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ee2da889-3df3-4db7-abf0-9f5a75174e96-plugins-config-volume\") on node \"ip-10-0-139-174.ec2.internal\" DevicePath \"\"" Sep 10 13:34:10.430974 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:34:10.430835 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/287603f3-efd3-46a9-bbfe-312f96974f76-workload-certs\") pod \"e2e-mt-7f8babf4-openshift-default-74f9755b6d-dzdc9\" (UID: \"287603f3-efd3-46a9-bbfe-312f96974f76\") " pod="openshift-ingress/e2e-mt-7f8babf4-openshift-default-74f9755b6d-dzdc9" Sep 10 13:34:10.430974 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:34:10.430876 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/287603f3-efd3-46a9-bbfe-312f96974f76-credential-socket\") pod \"e2e-mt-7f8babf4-openshift-default-74f9755b6d-dzdc9\" (UID: \"287603f3-efd3-46a9-bbfe-312f96974f76\") " pod="openshift-ingress/e2e-mt-7f8babf4-openshift-default-74f9755b6d-dzdc9" Sep 10 13:34:10.431528 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:34:10.431333 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/287603f3-efd3-46a9-bbfe-312f96974f76-istiod-ca-cert\") pod \"e2e-mt-7f8babf4-openshift-default-74f9755b6d-dzdc9\" (UID: \"287603f3-efd3-46a9-bbfe-312f96974f76\") " pod="openshift-ingress/e2e-mt-7f8babf4-openshift-default-74f9755b6d-dzdc9" Sep 10 13:34:10.431701 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:34:10.431630 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/287603f3-efd3-46a9-bbfe-312f96974f76-istio-data\") pod \"e2e-mt-7f8babf4-openshift-default-74f9755b6d-dzdc9\" (UID: \"287603f3-efd3-46a9-bbfe-312f96974f76\") " pod="openshift-ingress/e2e-mt-7f8babf4-openshift-default-74f9755b6d-dzdc9" Sep 10 13:34:10.432796 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:34:10.432770 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/516a2551-a59b-4435-8b6b-fdb75ca4d1d5-kube-api-access-zf8xl" (OuterVolumeSpecName: "kube-api-access-zf8xl") pod "516a2551-a59b-4435-8b6b-fdb75ca4d1d5" (UID: "516a2551-a59b-4435-8b6b-fdb75ca4d1d5"). InnerVolumeSpecName "kube-api-access-zf8xl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 10 13:34:10.433601 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:34:10.433556 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/287603f3-efd3-46a9-bbfe-312f96974f76-istio-envoy\") pod \"e2e-mt-7f8babf4-openshift-default-74f9755b6d-dzdc9\" (UID: \"287603f3-efd3-46a9-bbfe-312f96974f76\") " pod="openshift-ingress/e2e-mt-7f8babf4-openshift-default-74f9755b6d-dzdc9" Sep 10 13:34:10.433601 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:34:10.433595 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/287603f3-efd3-46a9-bbfe-312f96974f76-istio-podinfo\") pod \"e2e-mt-7f8babf4-openshift-default-74f9755b6d-dzdc9\" (UID: \"287603f3-efd3-46a9-bbfe-312f96974f76\") " pod="openshift-ingress/e2e-mt-7f8babf4-openshift-default-74f9755b6d-dzdc9" Sep 10 13:34:10.433803 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:34:10.433604 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ee2da889-3df3-4db7-abf0-9f5a75174e96-kube-api-access-5djkx" (OuterVolumeSpecName: "kube-api-access-5djkx") pod "ee2da889-3df3-4db7-abf0-9f5a75174e96" (UID: "ee2da889-3df3-4db7-abf0-9f5a75174e96"). InnerVolumeSpecName "kube-api-access-5djkx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 10 13:34:10.442502 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:34:10.442474 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/287603f3-efd3-46a9-bbfe-312f96974f76-istio-token\") pod \"e2e-mt-7f8babf4-openshift-default-74f9755b6d-dzdc9\" (UID: \"287603f3-efd3-46a9-bbfe-312f96974f76\") " pod="openshift-ingress/e2e-mt-7f8babf4-openshift-default-74f9755b6d-dzdc9" Sep 10 13:34:10.442624 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:34:10.442571 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qn5rh\" (UniqueName: \"kubernetes.io/projected/287603f3-efd3-46a9-bbfe-312f96974f76-kube-api-access-qn5rh\") pod \"e2e-mt-7f8babf4-openshift-default-74f9755b6d-dzdc9\" (UID: \"287603f3-efd3-46a9-bbfe-312f96974f76\") " pod="openshift-ingress/e2e-mt-7f8babf4-openshift-default-74f9755b6d-dzdc9" Sep 10 13:34:10.514030 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:34:10.513958 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-7f8babf4-openshift-default-74f9755b6d-dzdc9" Sep 10 13:34:10.531212 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:34:10.531185 2557 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zf8xl\" (UniqueName: \"kubernetes.io/projected/516a2551-a59b-4435-8b6b-fdb75ca4d1d5-kube-api-access-zf8xl\") on node \"ip-10-0-139-174.ec2.internal\" DevicePath \"\"" Sep 10 13:34:10.531301 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:34:10.531219 2557 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5djkx\" (UniqueName: \"kubernetes.io/projected/ee2da889-3df3-4db7-abf0-9f5a75174e96-kube-api-access-5djkx\") on node \"ip-10-0-139-174.ec2.internal\" DevicePath \"\"" Sep 10 13:34:10.807834 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:34:10.806016 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-7f8babf4-openshift-default-74f9755b6d-dzdc9"] Sep 10 13:34:10.808874 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:34:10.808824 2557 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892152Ki","pods":"250"} Sep 10 13:34:10.809000 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:34:10.808962 2557 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892152Ki","pods":"250"} Sep 10 13:34:10.809303 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:34:10.809271 2557 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892152Ki","pods":"250"} Sep 10 13:34:10.941258 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:34:10.941223 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-7f8babf4-openshift-default-74f9755b6d-dzdc9" event={"ID":"287603f3-efd3-46a9-bbfe-312f96974f76","Type":"ContainerStarted","Data":"7271da9c01225af01e6db751f4c8a293d3d8f17c252e0722d26916d6c4469885"} Sep 10 13:34:10.941467 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:34:10.941288 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-7f8babf4-openshift-default-74f9755b6d-dzdc9" event={"ID":"287603f3-efd3-46a9-bbfe-312f96974f76","Type":"ContainerStarted","Data":"3cf472f37453b04ddf18099305582232e361d016f702d73dec0830ab36ea23f1"} Sep 10 13:34:10.942848 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:34:10.942821 2557 generic.go:356] "Generic (PLEG): container finished" podID="ee2da889-3df3-4db7-abf0-9f5a75174e96" containerID="d531547914656f52029fd7af93bcbf2e6fb284576964283c0860da3bf39372d1" exitCode=0 Sep 10 13:34:10.942975 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:34:10.942851 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-39df6e-7bdcf56986-9czvg" event={"ID":"ee2da889-3df3-4db7-abf0-9f5a75174e96","Type":"ContainerDied","Data":"d531547914656f52029fd7af93bcbf2e6fb284576964283c0860da3bf39372d1"} Sep 10 13:34:10.942975 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:34:10.942885 2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-39df6e-7bdcf56986-9czvg" Sep 10 13:34:10.942975 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:34:10.942902 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-39df6e-7bdcf56986-9czvg" event={"ID":"ee2da889-3df3-4db7-abf0-9f5a75174e96","Type":"ContainerDied","Data":"240a7e80a1cebc31cdaaf7ef4f91a8d3479a48fffeacf7076fe6dc0793a4bd7f"} Sep 10 13:34:10.942975 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:34:10.942937 2557 scope.go:117] "RemoveContainer" containerID="d531547914656f52029fd7af93bcbf2e6fb284576964283c0860da3bf39372d1" Sep 10 13:34:10.944692 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:34:10.944654 2557 generic.go:356] "Generic (PLEG): container finished" podID="516a2551-a59b-4435-8b6b-fdb75ca4d1d5" containerID="7fa801e40c5a227a247bd78d5d448ba094bf0c1331f94dee6b65d6bf262d177e" exitCode=0 Sep 10 13:34:10.944816 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:34:10.944752 2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-39df6e-758db9c89f-d6wdl" Sep 10 13:34:10.944938 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:34:10.944841 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-39df6e-758db9c89f-d6wdl" event={"ID":"516a2551-a59b-4435-8b6b-fdb75ca4d1d5","Type":"ContainerDied","Data":"7fa801e40c5a227a247bd78d5d448ba094bf0c1331f94dee6b65d6bf262d177e"} Sep 10 13:34:10.944938 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:34:10.944878 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-39df6e-758db9c89f-d6wdl" event={"ID":"516a2551-a59b-4435-8b6b-fdb75ca4d1d5","Type":"ContainerDied","Data":"ca15db838c2d527ccf094fbd403bf75dc8718e21fc46329d691db84f6362909a"} Sep 10 13:34:10.953970 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:34:10.953949 2557 scope.go:117] "RemoveContainer" containerID="d531547914656f52029fd7af93bcbf2e6fb284576964283c0860da3bf39372d1" Sep 10 13:34:10.954295 ip-10-0-139-174 kubenswrapper[2557]: E0910 13:34:10.954273 2557 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d531547914656f52029fd7af93bcbf2e6fb284576964283c0860da3bf39372d1\": container with ID starting with d531547914656f52029fd7af93bcbf2e6fb284576964283c0860da3bf39372d1 not found: ID does not exist" containerID="d531547914656f52029fd7af93bcbf2e6fb284576964283c0860da3bf39372d1" Sep 10 13:34:10.954366 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:34:10.954307 2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d531547914656f52029fd7af93bcbf2e6fb284576964283c0860da3bf39372d1"} err="failed to get container status \"d531547914656f52029fd7af93bcbf2e6fb284576964283c0860da3bf39372d1\": rpc error: code = NotFound desc = could not find container \"d531547914656f52029fd7af93bcbf2e6fb284576964283c0860da3bf39372d1\": container with ID starting with d531547914656f52029fd7af93bcbf2e6fb284576964283c0860da3bf39372d1 not found: ID does not exist" Sep 10 13:34:10.954366 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:34:10.954331 2557 scope.go:117] "RemoveContainer" containerID="7fa801e40c5a227a247bd78d5d448ba094bf0c1331f94dee6b65d6bf262d177e" Sep 10 13:34:10.964312 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:34:10.964281 2557 scope.go:117] "RemoveContainer" containerID="7fa801e40c5a227a247bd78d5d448ba094bf0c1331f94dee6b65d6bf262d177e" Sep 10 13:34:10.965117 ip-10-0-139-174 kubenswrapper[2557]: E0910 13:34:10.964676 2557 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7fa801e40c5a227a247bd78d5d448ba094bf0c1331f94dee6b65d6bf262d177e\": container with ID starting with 7fa801e40c5a227a247bd78d5d448ba094bf0c1331f94dee6b65d6bf262d177e not found: ID does not exist" containerID="7fa801e40c5a227a247bd78d5d448ba094bf0c1331f94dee6b65d6bf262d177e" Sep 10 13:34:10.965117 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:34:10.964709 2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7fa801e40c5a227a247bd78d5d448ba094bf0c1331f94dee6b65d6bf262d177e"} err="failed to get container status \"7fa801e40c5a227a247bd78d5d448ba094bf0c1331f94dee6b65d6bf262d177e\": rpc error: code = NotFound desc = could not find container \"7fa801e40c5a227a247bd78d5d448ba094bf0c1331f94dee6b65d6bf262d177e\": container with ID starting with 7fa801e40c5a227a247bd78d5d448ba094bf0c1331f94dee6b65d6bf262d177e not found: ID does not exist" Sep 10 13:34:10.965117 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:34:10.965047 2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-7f8babf4-openshift-default-74f9755b6d-dzdc9" podStartSLOduration=0.965030588 podStartE2EDuration="965.030588ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 13:34:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 13:34:10.964023093 +0000 UTC m=+1011.271017509" watchObservedRunningTime="2026-09-10 13:34:10.965030588 +0000 UTC m=+1011.272025003" Sep 10 13:34:10.981329 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:34:10.981305 2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-39df6e-7bdcf56986-9czvg"] Sep 10 13:34:10.987398 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:34:10.987373 2557 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-39df6e-7bdcf56986-9czvg"] Sep 10 13:34:10.997402 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:34:10.997375 2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-39df6e-758db9c89f-d6wdl"] Sep 10 13:34:11.003886 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:34:11.003866 2557 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-39df6e-758db9c89f-d6wdl"] Sep 10 13:34:11.514251 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:34:11.514224 2557 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-7f8babf4-openshift-default-74f9755b6d-dzdc9" Sep 10 13:34:11.519105 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:34:11.519080 2557 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-7f8babf4-openshift-default-74f9755b6d-dzdc9" Sep 10 13:34:11.953338 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:34:11.953308 2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-7f8babf4-openshift-default-74f9755b6d-dzdc9" Sep 10 13:34:11.954297 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:34:11.954280 2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-7f8babf4-openshift-default-74f9755b6d-dzdc9" Sep 10 13:34:12.220779 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:34:12.220689 2557 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="516a2551-a59b-4435-8b6b-fdb75ca4d1d5" path="/var/lib/kubelet/pods/516a2551-a59b-4435-8b6b-fdb75ca4d1d5/volumes" Sep 10 13:34:12.221161 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:34:12.221047 2557 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ee2da889-3df3-4db7-abf0-9f5a75174e96" path="/var/lib/kubelet/pods/ee2da889-3df3-4db7-abf0-9f5a75174e96/volumes" Sep 10 13:34:13.123948 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:34:13.123904 2557 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-inf-a-39df6e/test-model-39df6e-kserve-698d88cd99-xgkz7" podUID="76f7bb42-a548-41cf-9bea-0f2734d74b24" containerName="main" containerID="cri-o://61db300f7ea9a655270b3da030c6f4a3bdf5dacb278c43c30fdbc91a06d685cb" gracePeriod=45 Sep 10 13:34:13.377764 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:34:13.377700 2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-39df6e/test-model-39df6e-kserve-698d88cd99-xgkz7" Sep 10 13:34:13.459548 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:34:13.459520 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/76f7bb42-a548-41cf-9bea-0f2734d74b24-tmp-dir\") pod \"76f7bb42-a548-41cf-9bea-0f2734d74b24\" (UID: \"76f7bb42-a548-41cf-9bea-0f2734d74b24\") " Sep 10 13:34:13.459712 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:34:13.459557 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/76f7bb42-a548-41cf-9bea-0f2734d74b24-tls-certs\") pod \"76f7bb42-a548-41cf-9bea-0f2734d74b24\" (UID: \"76f7bb42-a548-41cf-9bea-0f2734d74b24\") " Sep 10 13:34:13.459712 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:34:13.459576 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xg628\" (UniqueName: \"kubernetes.io/projected/76f7bb42-a548-41cf-9bea-0f2734d74b24-kube-api-access-xg628\") pod \"76f7bb42-a548-41cf-9bea-0f2734d74b24\" (UID: \"76f7bb42-a548-41cf-9bea-0f2734d74b24\") " Sep 10 13:34:13.459712 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:34:13.459613 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/76f7bb42-a548-41cf-9bea-0f2734d74b24-dshm\") pod \"76f7bb42-a548-41cf-9bea-0f2734d74b24\" (UID: \"76f7bb42-a548-41cf-9bea-0f2734d74b24\") " Sep 10 13:34:13.459712 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:34:13.459674 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/76f7bb42-a548-41cf-9bea-0f2734d74b24-home\") pod \"76f7bb42-a548-41cf-9bea-0f2734d74b24\" (UID: \"76f7bb42-a548-41cf-9bea-0f2734d74b24\") " Sep 10 13:34:13.459712 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:34:13.459711 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/76f7bb42-a548-41cf-9bea-0f2734d74b24-model-cache\") pod \"76f7bb42-a548-41cf-9bea-0f2734d74b24\" (UID: \"76f7bb42-a548-41cf-9bea-0f2734d74b24\") " Sep 10 13:34:13.459994 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:34:13.459834 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/76f7bb42-a548-41cf-9bea-0f2734d74b24-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "76f7bb42-a548-41cf-9bea-0f2734d74b24" (UID: "76f7bb42-a548-41cf-9bea-0f2734d74b24"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 13:34:13.460054 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:34:13.459996 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/76f7bb42-a548-41cf-9bea-0f2734d74b24-home" (OuterVolumeSpecName: "home") pod "76f7bb42-a548-41cf-9bea-0f2734d74b24" (UID: "76f7bb42-a548-41cf-9bea-0f2734d74b24"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 13:34:13.460054 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:34:13.460019 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/76f7bb42-a548-41cf-9bea-0f2734d74b24-model-cache" (OuterVolumeSpecName: "model-cache") pod "76f7bb42-a548-41cf-9bea-0f2734d74b24" (UID: "76f7bb42-a548-41cf-9bea-0f2734d74b24"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 13:34:13.460177 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:34:13.460159 2557 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/76f7bb42-a548-41cf-9bea-0f2734d74b24-model-cache\") on node \"ip-10-0-139-174.ec2.internal\" DevicePath \"\"" Sep 10 13:34:13.460218 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:34:13.460187 2557 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/76f7bb42-a548-41cf-9bea-0f2734d74b24-tmp-dir\") on node \"ip-10-0-139-174.ec2.internal\" DevicePath \"\"" Sep 10 13:34:13.460218 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:34:13.460200 2557 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/76f7bb42-a548-41cf-9bea-0f2734d74b24-home\") on node \"ip-10-0-139-174.ec2.internal\" DevicePath \"\"" Sep 10 13:34:13.461920 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:34:13.461893 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/76f7bb42-a548-41cf-9bea-0f2734d74b24-dshm" (OuterVolumeSpecName: "dshm") pod "76f7bb42-a548-41cf-9bea-0f2734d74b24" (UID: "76f7bb42-a548-41cf-9bea-0f2734d74b24"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 13:34:13.462030 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:34:13.461922 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/76f7bb42-a548-41cf-9bea-0f2734d74b24-kube-api-access-xg628" (OuterVolumeSpecName: "kube-api-access-xg628") pod "76f7bb42-a548-41cf-9bea-0f2734d74b24" (UID: "76f7bb42-a548-41cf-9bea-0f2734d74b24"). InnerVolumeSpecName "kube-api-access-xg628". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 10 13:34:13.462030 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:34:13.461948 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/76f7bb42-a548-41cf-9bea-0f2734d74b24-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "76f7bb42-a548-41cf-9bea-0f2734d74b24" (UID: "76f7bb42-a548-41cf-9bea-0f2734d74b24"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 10 13:34:13.561543 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:34:13.561520 2557 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/76f7bb42-a548-41cf-9bea-0f2734d74b24-tls-certs\") on node \"ip-10-0-139-174.ec2.internal\" DevicePath \"\"" Sep 10 13:34:13.561543 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:34:13.561542 2557 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xg628\" (UniqueName: \"kubernetes.io/projected/76f7bb42-a548-41cf-9bea-0f2734d74b24-kube-api-access-xg628\") on node \"ip-10-0-139-174.ec2.internal\" DevicePath \"\"" Sep 10 13:34:13.561676 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:34:13.561551 2557 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/76f7bb42-a548-41cf-9bea-0f2734d74b24-dshm\") on node \"ip-10-0-139-174.ec2.internal\" DevicePath \"\"" Sep 10 13:34:13.969140 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:34:13.969109 2557 generic.go:356] "Generic (PLEG): container finished" podID="76f7bb42-a548-41cf-9bea-0f2734d74b24" containerID="61db300f7ea9a655270b3da030c6f4a3bdf5dacb278c43c30fdbc91a06d685cb" exitCode=0 Sep 10 13:34:13.969278 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:34:13.969153 2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-39df6e/test-model-39df6e-kserve-698d88cd99-xgkz7" Sep 10 13:34:13.969278 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:34:13.969198 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-39df6e/test-model-39df6e-kserve-698d88cd99-xgkz7" event={"ID":"76f7bb42-a548-41cf-9bea-0f2734d74b24","Type":"ContainerDied","Data":"61db300f7ea9a655270b3da030c6f4a3bdf5dacb278c43c30fdbc91a06d685cb"} Sep 10 13:34:13.969278 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:34:13.969240 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-39df6e/test-model-39df6e-kserve-698d88cd99-xgkz7" event={"ID":"76f7bb42-a548-41cf-9bea-0f2734d74b24","Type":"ContainerDied","Data":"a7e02e56aff7af454a121d1347ec75589a2a431a02cd266be77eaa5704a175c0"} Sep 10 13:34:13.969278 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:34:13.969265 2557 scope.go:117] "RemoveContainer" containerID="61db300f7ea9a655270b3da030c6f4a3bdf5dacb278c43c30fdbc91a06d685cb" Sep 10 13:34:13.981448 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:34:13.981413 2557 scope.go:117] "RemoveContainer" containerID="61db300f7ea9a655270b3da030c6f4a3bdf5dacb278c43c30fdbc91a06d685cb" Sep 10 13:34:13.981829 ip-10-0-139-174 kubenswrapper[2557]: E0910 13:34:13.981807 2557 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"61db300f7ea9a655270b3da030c6f4a3bdf5dacb278c43c30fdbc91a06d685cb\": container with ID starting with 61db300f7ea9a655270b3da030c6f4a3bdf5dacb278c43c30fdbc91a06d685cb not found: ID does not exist" containerID="61db300f7ea9a655270b3da030c6f4a3bdf5dacb278c43c30fdbc91a06d685cb" Sep 10 13:34:13.981903 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:34:13.981838 2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"61db300f7ea9a655270b3da030c6f4a3bdf5dacb278c43c30fdbc91a06d685cb"} err="failed to get container status \"61db300f7ea9a655270b3da030c6f4a3bdf5dacb278c43c30fdbc91a06d685cb\": rpc error: code = NotFound desc = could not find container \"61db300f7ea9a655270b3da030c6f4a3bdf5dacb278c43c30fdbc91a06d685cb\": container with ID starting with 61db300f7ea9a655270b3da030c6f4a3bdf5dacb278c43c30fdbc91a06d685cb not found: ID does not exist" Sep 10 13:34:14.015634 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:34:14.015609 2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-39df6e/test-model-39df6e-kserve-698d88cd99-xgkz7"] Sep 10 13:34:14.024441 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:34:14.024403 2557 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-a-39df6e/test-model-39df6e-kserve-698d88cd99-xgkz7"] Sep 10 13:34:14.220538 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:34:14.220460 2557 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="76f7bb42-a548-41cf-9bea-0f2734d74b24" path="/var/lib/kubelet/pods/76f7bb42-a548-41cf-9bea-0f2734d74b24/volumes" Sep 10 13:34:15.149908 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:34:15.149867 2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-7f8babf4-6c88b456b5-c8bpr"] Sep 10 13:34:15.151024 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:34:15.150987 2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="516a2551-a59b-4435-8b6b-fdb75ca4d1d5" containerName="payload-processing" Sep 10 13:34:15.151024 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:34:15.151021 2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="516a2551-a59b-4435-8b6b-fdb75ca4d1d5" containerName="payload-processing" Sep 10 13:34:15.151204 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:34:15.151044 2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="76f7bb42-a548-41cf-9bea-0f2734d74b24" containerName="main" Sep 10 13:34:15.151204 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:34:15.151054 2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="76f7bb42-a548-41cf-9bea-0f2734d74b24" containerName="main" Sep 10 13:34:15.151204 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:34:15.151088 2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ee2da889-3df3-4db7-abf0-9f5a75174e96" containerName="payload-pre-processing" Sep 10 13:34:15.151204 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:34:15.151097 2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="ee2da889-3df3-4db7-abf0-9f5a75174e96" containerName="payload-pre-processing" Sep 10 13:34:15.151204 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:34:15.151185 2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="ee2da889-3df3-4db7-abf0-9f5a75174e96" containerName="payload-pre-processing" Sep 10 13:34:15.151204 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:34:15.151198 2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="516a2551-a59b-4435-8b6b-fdb75ca4d1d5" containerName="payload-processing" Sep 10 13:34:15.151204 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:34:15.151206 2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="76f7bb42-a548-41cf-9bea-0f2734d74b24" containerName="main" Sep 10 13:34:15.156164 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:34:15.156129 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-7f8babf4-6c88b456b5-c8bpr" Sep 10 13:34:15.158940 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:34:15.158921 2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-mt-7f8babf4\"" Sep 10 13:34:15.159189 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:34:15.159157 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-mt-7f8babf4-dockercfg-qgqxk\"" Sep 10 13:34:15.174698 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:34:15.174678 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-7f8babf4-6c88b456b5-c8bpr"] Sep 10 13:34:15.279174 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:34:15.279144 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w9xxn\" (UniqueName: \"kubernetes.io/projected/3da88b7f-bbfb-4521-acc5-dfe20cacbced-kube-api-access-w9xxn\") pod \"payload-processing-e2e-mt-7f8babf4-6c88b456b5-c8bpr\" (UID: \"3da88b7f-bbfb-4521-acc5-dfe20cacbced\") " pod="openshift-ingress/payload-processing-e2e-mt-7f8babf4-6c88b456b5-c8bpr" Sep 10 13:34:15.279339 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:34:15.279230 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3da88b7f-bbfb-4521-acc5-dfe20cacbced-plugins-config-volume\") pod \"payload-processing-e2e-mt-7f8babf4-6c88b456b5-c8bpr\" (UID: \"3da88b7f-bbfb-4521-acc5-dfe20cacbced\") " pod="openshift-ingress/payload-processing-e2e-mt-7f8babf4-6c88b456b5-c8bpr" Sep 10 13:34:15.380576 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:34:15.380539 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w9xxn\" (UniqueName: \"kubernetes.io/projected/3da88b7f-bbfb-4521-acc5-dfe20cacbced-kube-api-access-w9xxn\") pod \"payload-processing-e2e-mt-7f8babf4-6c88b456b5-c8bpr\" (UID: \"3da88b7f-bbfb-4521-acc5-dfe20cacbced\") " pod="openshift-ingress/payload-processing-e2e-mt-7f8babf4-6c88b456b5-c8bpr" Sep 10 13:34:15.380715 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:34:15.380651 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3da88b7f-bbfb-4521-acc5-dfe20cacbced-plugins-config-volume\") pod \"payload-processing-e2e-mt-7f8babf4-6c88b456b5-c8bpr\" (UID: \"3da88b7f-bbfb-4521-acc5-dfe20cacbced\") " pod="openshift-ingress/payload-processing-e2e-mt-7f8babf4-6c88b456b5-c8bpr" Sep 10 13:34:15.381289 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:34:15.381269 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3da88b7f-bbfb-4521-acc5-dfe20cacbced-plugins-config-volume\") pod \"payload-processing-e2e-mt-7f8babf4-6c88b456b5-c8bpr\" (UID: \"3da88b7f-bbfb-4521-acc5-dfe20cacbced\") " pod="openshift-ingress/payload-processing-e2e-mt-7f8babf4-6c88b456b5-c8bpr" Sep 10 13:34:15.399126 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:34:15.399106 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-w9xxn\" (UniqueName: \"kubernetes.io/projected/3da88b7f-bbfb-4521-acc5-dfe20cacbced-kube-api-access-w9xxn\") pod \"payload-processing-e2e-mt-7f8babf4-6c88b456b5-c8bpr\" (UID: \"3da88b7f-bbfb-4521-acc5-dfe20cacbced\") " pod="openshift-ingress/payload-processing-e2e-mt-7f8babf4-6c88b456b5-c8bpr" Sep 10 13:34:15.467125 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:34:15.467094 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-7f8babf4-6c88b456b5-c8bpr" Sep 10 13:34:15.754701 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:34:15.754673 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-7f8babf4-6c88b456b5-c8bpr"] Sep 10 13:34:15.756295 ip-10-0-139-174 kubenswrapper[2557]: W0910 13:34:15.756267 2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3da88b7f_bbfb_4521_acc5_dfe20cacbced.slice/crio-15fa18060f16452a74548b5d0081bccd0af185f8e8a30257102850d804ff95aa WatchSource:0}: Error finding container 15fa18060f16452a74548b5d0081bccd0af185f8e8a30257102850d804ff95aa: Status 404 returned error can't find the container with id 15fa18060f16452a74548b5d0081bccd0af185f8e8a30257102850d804ff95aa Sep 10 13:34:15.985608 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:34:15.985566 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-7f8babf4-6c88b456b5-c8bpr" event={"ID":"3da88b7f-bbfb-4521-acc5-dfe20cacbced","Type":"ContainerStarted","Data":"fb83f56d7a4b17a9ea1ea0a3068fbe340d78d6d357beb3a8b4305b671b5e360e"} Sep 10 13:34:15.985809 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:34:15.985638 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-7f8babf4-6c88b456b5-c8bpr" event={"ID":"3da88b7f-bbfb-4521-acc5-dfe20cacbced","Type":"ContainerStarted","Data":"15fa18060f16452a74548b5d0081bccd0af185f8e8a30257102850d804ff95aa"} Sep 10 13:34:15.985809 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:34:15.985723 2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-mt-7f8babf4-6c88b456b5-c8bpr" Sep 10 13:34:16.013760 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:34:16.013666 2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-mt-7f8babf4-6c88b456b5-c8bpr" podStartSLOduration=1.013651234 podStartE2EDuration="1.013651234s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 13:34:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 13:34:16.012226343 +0000 UTC m=+1016.319220770" watchObservedRunningTime="2026-09-10 13:34:16.013651234 +0000 UTC m=+1016.320645647" Sep 10 13:34:19.024843 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:34:19.024809 2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-39df6e-openshift-default-69b8d58b68-cpt89"] Sep 10 13:34:19.025681 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:34:19.025625 2557 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-39df6e-openshift-default-69b8d58b68-cpt89" podUID="bbab6794-a024-4b72-a8f2-2b9a4e85a3f2" containerName="istio-proxy" containerID="cri-o://750b8fc01729c33da6eb4abe1fdfed8db3dca6ba4bcb4e103f01be459caeb32a" gracePeriod=30 Sep 10 13:34:24.334035 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:34:24.334001 2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-39df6e-openshift-default-69b8d58b68-cpt89" Sep 10 13:34:24.392120 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:34:24.392081 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-zn9z4\" (UniqueName: \"kubernetes.io/projected/bbab6794-a024-4b72-a8f2-2b9a4e85a3f2-kube-api-access-zn9z4\") pod \"bbab6794-a024-4b72-a8f2-2b9a4e85a3f2\" (UID: \"bbab6794-a024-4b72-a8f2-2b9a4e85a3f2\") " Sep 10 13:34:24.392325 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:34:24.392145 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bbab6794-a024-4b72-a8f2-2b9a4e85a3f2-istio-envoy\") pod \"bbab6794-a024-4b72-a8f2-2b9a4e85a3f2\" (UID: \"bbab6794-a024-4b72-a8f2-2b9a4e85a3f2\") " Sep 10 13:34:24.392325 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:34:24.392178 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bbab6794-a024-4b72-a8f2-2b9a4e85a3f2-credential-socket\") pod \"bbab6794-a024-4b72-a8f2-2b9a4e85a3f2\" (UID: \"bbab6794-a024-4b72-a8f2-2b9a4e85a3f2\") " Sep 10 13:34:24.392325 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:34:24.392224 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bbab6794-a024-4b72-a8f2-2b9a4e85a3f2-istio-podinfo\") pod \"bbab6794-a024-4b72-a8f2-2b9a4e85a3f2\" (UID: \"bbab6794-a024-4b72-a8f2-2b9a4e85a3f2\") " Sep 10 13:34:24.392325 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:34:24.392254 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bbab6794-a024-4b72-a8f2-2b9a4e85a3f2-istiod-ca-cert\") pod \"bbab6794-a024-4b72-a8f2-2b9a4e85a3f2\" (UID: \"bbab6794-a024-4b72-a8f2-2b9a4e85a3f2\") " Sep 10 13:34:24.392325 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:34:24.392281 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bbab6794-a024-4b72-a8f2-2b9a4e85a3f2-workload-certs\") pod \"bbab6794-a024-4b72-a8f2-2b9a4e85a3f2\" (UID: \"bbab6794-a024-4b72-a8f2-2b9a4e85a3f2\") " Sep 10 13:34:24.392643 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:34:24.392334 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bbab6794-a024-4b72-a8f2-2b9a4e85a3f2-workload-socket\") pod \"bbab6794-a024-4b72-a8f2-2b9a4e85a3f2\" (UID: \"bbab6794-a024-4b72-a8f2-2b9a4e85a3f2\") " Sep 10 13:34:24.392643 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:34:24.392391 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bbab6794-a024-4b72-a8f2-2b9a4e85a3f2-istio-token\") pod \"bbab6794-a024-4b72-a8f2-2b9a4e85a3f2\" (UID: \"bbab6794-a024-4b72-a8f2-2b9a4e85a3f2\") " Sep 10 13:34:24.392643 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:34:24.392458 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bbab6794-a024-4b72-a8f2-2b9a4e85a3f2-istio-data\") pod \"bbab6794-a024-4b72-a8f2-2b9a4e85a3f2\" (UID: \"bbab6794-a024-4b72-a8f2-2b9a4e85a3f2\") " Sep 10 13:34:24.392643 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:34:24.392521 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bbab6794-a024-4b72-a8f2-2b9a4e85a3f2-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "bbab6794-a024-4b72-a8f2-2b9a4e85a3f2" (UID: "bbab6794-a024-4b72-a8f2-2b9a4e85a3f2"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 13:34:24.392866 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:34:24.392726 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bbab6794-a024-4b72-a8f2-2b9a4e85a3f2-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "bbab6794-a024-4b72-a8f2-2b9a4e85a3f2" (UID: "bbab6794-a024-4b72-a8f2-2b9a4e85a3f2"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 13:34:24.392866 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:34:24.392749 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bbab6794-a024-4b72-a8f2-2b9a4e85a3f2-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "bbab6794-a024-4b72-a8f2-2b9a4e85a3f2" (UID: "bbab6794-a024-4b72-a8f2-2b9a4e85a3f2"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 10 13:34:24.392866 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:34:24.392854 2557 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bbab6794-a024-4b72-a8f2-2b9a4e85a3f2-credential-socket\") on node \"ip-10-0-139-174.ec2.internal\" DevicePath \"\"" Sep 10 13:34:24.393050 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:34:24.392876 2557 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bbab6794-a024-4b72-a8f2-2b9a4e85a3f2-istiod-ca-cert\") on node \"ip-10-0-139-174.ec2.internal\" DevicePath \"\"" Sep 10 13:34:24.393050 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:34:24.392890 2557 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bbab6794-a024-4b72-a8f2-2b9a4e85a3f2-workload-socket\") on node \"ip-10-0-139-174.ec2.internal\" DevicePath \"\"" Sep 10 13:34:24.393050 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:34:24.392949 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bbab6794-a024-4b72-a8f2-2b9a4e85a3f2-istio-data" (OuterVolumeSpecName: "istio-data") pod "bbab6794-a024-4b72-a8f2-2b9a4e85a3f2" (UID: "bbab6794-a024-4b72-a8f2-2b9a4e85a3f2"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 13:34:24.393208 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:34:24.393104 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bbab6794-a024-4b72-a8f2-2b9a4e85a3f2-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "bbab6794-a024-4b72-a8f2-2b9a4e85a3f2" (UID: "bbab6794-a024-4b72-a8f2-2b9a4e85a3f2"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 13:34:24.394796 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:34:24.394539 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bbab6794-a024-4b72-a8f2-2b9a4e85a3f2-kube-api-access-zn9z4" (OuterVolumeSpecName: "kube-api-access-zn9z4") pod "bbab6794-a024-4b72-a8f2-2b9a4e85a3f2" (UID: "bbab6794-a024-4b72-a8f2-2b9a4e85a3f2"). InnerVolumeSpecName "kube-api-access-zn9z4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 10 13:34:24.395175 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:34:24.395148 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/bbab6794-a024-4b72-a8f2-2b9a4e85a3f2-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "bbab6794-a024-4b72-a8f2-2b9a4e85a3f2" (UID: "bbab6794-a024-4b72-a8f2-2b9a4e85a3f2"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Sep 10 13:34:24.395482 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:34:24.395461 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bbab6794-a024-4b72-a8f2-2b9a4e85a3f2-istio-token" (OuterVolumeSpecName: "istio-token") pod "bbab6794-a024-4b72-a8f2-2b9a4e85a3f2" (UID: "bbab6794-a024-4b72-a8f2-2b9a4e85a3f2"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 10 13:34:24.395767 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:34:24.395752 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bbab6794-a024-4b72-a8f2-2b9a4e85a3f2-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "bbab6794-a024-4b72-a8f2-2b9a4e85a3f2" (UID: "bbab6794-a024-4b72-a8f2-2b9a4e85a3f2"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 13:34:24.494062 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:34:24.494037 2557 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bbab6794-a024-4b72-a8f2-2b9a4e85a3f2-istio-podinfo\") on node \"ip-10-0-139-174.ec2.internal\" DevicePath \"\"" Sep 10 13:34:24.494062 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:34:24.494063 2557 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bbab6794-a024-4b72-a8f2-2b9a4e85a3f2-workload-certs\") on node \"ip-10-0-139-174.ec2.internal\" DevicePath \"\"" Sep 10 13:34:24.494210 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:34:24.494077 2557 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bbab6794-a024-4b72-a8f2-2b9a4e85a3f2-istio-token\") on node \"ip-10-0-139-174.ec2.internal\" DevicePath \"\"" Sep 10 13:34:24.494210 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:34:24.494085 2557 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bbab6794-a024-4b72-a8f2-2b9a4e85a3f2-istio-data\") on node \"ip-10-0-139-174.ec2.internal\" DevicePath \"\"" Sep 10 13:34:24.494210 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:34:24.494092 2557 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zn9z4\" (UniqueName: \"kubernetes.io/projected/bbab6794-a024-4b72-a8f2-2b9a4e85a3f2-kube-api-access-zn9z4\") on node \"ip-10-0-139-174.ec2.internal\" DevicePath \"\"" Sep 10 13:34:24.494210 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:34:24.494101 2557 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bbab6794-a024-4b72-a8f2-2b9a4e85a3f2-istio-envoy\") on node \"ip-10-0-139-174.ec2.internal\" DevicePath \"\"" Sep 10 13:34:25.046174 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:34:25.046140 2557 generic.go:356] "Generic (PLEG): container finished" podID="bbab6794-a024-4b72-a8f2-2b9a4e85a3f2" containerID="750b8fc01729c33da6eb4abe1fdfed8db3dca6ba4bcb4e103f01be459caeb32a" exitCode=0 Sep 10 13:34:25.046358 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:34:25.046199 2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-39df6e-openshift-default-69b8d58b68-cpt89" Sep 10 13:34:25.046358 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:34:25.046220 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-39df6e-openshift-default-69b8d58b68-cpt89" event={"ID":"bbab6794-a024-4b72-a8f2-2b9a4e85a3f2","Type":"ContainerDied","Data":"750b8fc01729c33da6eb4abe1fdfed8db3dca6ba4bcb4e103f01be459caeb32a"} Sep 10 13:34:25.046358 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:34:25.046281 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-39df6e-openshift-default-69b8d58b68-cpt89" event={"ID":"bbab6794-a024-4b72-a8f2-2b9a4e85a3f2","Type":"ContainerDied","Data":"a514481c2e8452b14b76e4b5abe02f6bc6c4479d3bba35c20e92f120072df982"} Sep 10 13:34:25.046358 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:34:25.046315 2557 scope.go:117] "RemoveContainer" containerID="750b8fc01729c33da6eb4abe1fdfed8db3dca6ba4bcb4e103f01be459caeb32a" Sep 10 13:34:25.056117 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:34:25.056098 2557 scope.go:117] "RemoveContainer" containerID="1a7c02d08463ced64d15e6b76d7168971c3ae490800a4ff832023f1f3a88dfaf" Sep 10 13:34:25.065067 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:34:25.065048 2557 scope.go:117] "RemoveContainer" containerID="750b8fc01729c33da6eb4abe1fdfed8db3dca6ba4bcb4e103f01be459caeb32a" Sep 10 13:34:25.065363 ip-10-0-139-174 kubenswrapper[2557]: E0910 13:34:25.065344 2557 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"750b8fc01729c33da6eb4abe1fdfed8db3dca6ba4bcb4e103f01be459caeb32a\": container with ID starting with 750b8fc01729c33da6eb4abe1fdfed8db3dca6ba4bcb4e103f01be459caeb32a not found: ID does not exist" containerID="750b8fc01729c33da6eb4abe1fdfed8db3dca6ba4bcb4e103f01be459caeb32a" Sep 10 13:34:25.065455 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:34:25.065371 2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"750b8fc01729c33da6eb4abe1fdfed8db3dca6ba4bcb4e103f01be459caeb32a"} err="failed to get container status \"750b8fc01729c33da6eb4abe1fdfed8db3dca6ba4bcb4e103f01be459caeb32a\": rpc error: code = NotFound desc = could not find container \"750b8fc01729c33da6eb4abe1fdfed8db3dca6ba4bcb4e103f01be459caeb32a\": container with ID starting with 750b8fc01729c33da6eb4abe1fdfed8db3dca6ba4bcb4e103f01be459caeb32a not found: ID does not exist" Sep 10 13:34:25.065455 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:34:25.065386 2557 scope.go:117] "RemoveContainer" containerID="1a7c02d08463ced64d15e6b76d7168971c3ae490800a4ff832023f1f3a88dfaf" Sep 10 13:34:25.065681 ip-10-0-139-174 kubenswrapper[2557]: E0910 13:34:25.065660 2557 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1a7c02d08463ced64d15e6b76d7168971c3ae490800a4ff832023f1f3a88dfaf\": container with ID starting with 1a7c02d08463ced64d15e6b76d7168971c3ae490800a4ff832023f1f3a88dfaf not found: ID does not exist" containerID="1a7c02d08463ced64d15e6b76d7168971c3ae490800a4ff832023f1f3a88dfaf" Sep 10 13:34:25.065737 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:34:25.065691 2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1a7c02d08463ced64d15e6b76d7168971c3ae490800a4ff832023f1f3a88dfaf"} err="failed to get container status \"1a7c02d08463ced64d15e6b76d7168971c3ae490800a4ff832023f1f3a88dfaf\": rpc error: code = NotFound desc = could not find container \"1a7c02d08463ced64d15e6b76d7168971c3ae490800a4ff832023f1f3a88dfaf\": container with ID starting with 1a7c02d08463ced64d15e6b76d7168971c3ae490800a4ff832023f1f3a88dfaf not found: ID does not exist" Sep 10 13:34:25.078623 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:34:25.078597 2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-39df6e-openshift-default-69b8d58b68-cpt89"] Sep 10 13:34:25.087167 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:34:25.087143 2557 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-39df6e-openshift-default-69b8d58b68-cpt89"] Sep 10 13:34:26.219957 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:34:26.219923 2557 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="bbab6794-a024-4b72-a8f2-2b9a4e85a3f2" path="/var/lib/kubelet/pods/bbab6794-a024-4b72-a8f2-2b9a4e85a3f2/volumes" Sep 10 13:34:26.994551 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:34:26.994524 2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-mt-7f8babf4-6c88b456b5-c8bpr" Sep 10 13:34:27.282462 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:34:27.282120 2557 patch_prober.go:28] interesting pod/e2e-mt-7f8babf4-openshift-default-74f9755b6d-dzdc9 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.61:15021/healthz/ready\": read tcp 10.134.0.2:59028->10.134.0.61:15021: read: connection reset by peer" start-of-body= Sep 10 13:34:27.282462 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:34:27.282212 2557 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-mt-7f8babf4-openshift-default-74f9755b6d-dzdc9" podUID="287603f3-efd3-46a9-bbfe-312f96974f76" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.61:15021/healthz/ready\": read tcp 10.134.0.2:59028->10.134.0.61:15021: read: connection reset by peer" Sep 10 13:34:28.073159 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:34:28.073129 2557 generic.go:356] "Generic (PLEG): container finished" podID="287603f3-efd3-46a9-bbfe-312f96974f76" containerID="7271da9c01225af01e6db751f4c8a293d3d8f17c252e0722d26916d6c4469885" exitCode=0 Sep 10 13:34:28.073348 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:34:28.073208 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-7f8babf4-openshift-default-74f9755b6d-dzdc9" event={"ID":"287603f3-efd3-46a9-bbfe-312f96974f76","Type":"ContainerDied","Data":"7271da9c01225af01e6db751f4c8a293d3d8f17c252e0722d26916d6c4469885"} Sep 10 13:34:28.073646 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:34:28.073629 2557 scope.go:117] "RemoveContainer" containerID="7271da9c01225af01e6db751f4c8a293d3d8f17c252e0722d26916d6c4469885" Sep 10 13:34:28.074700 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:34:28.074669 2557 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892152Ki","pods":"250"} Sep 10 13:34:28.074787 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:34:28.074760 2557 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892152Ki","pods":"250"} Sep 10 13:34:28.074824 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:34:28.074792 2557 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892152Ki","pods":"250"} Sep 10 13:34:29.080533 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:34:29.080502 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-7f8babf4-openshift-default-74f9755b6d-dzdc9" event={"ID":"287603f3-efd3-46a9-bbfe-312f96974f76","Type":"ContainerStarted","Data":"f279ee7e7781d02a864be4367ebee3bc23385711bfcaa5cacbebd6810c4d56d0"} Sep 10 13:34:29.514597 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:34:29.514563 2557 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-7f8babf4-openshift-default-74f9755b6d-dzdc9" Sep 10 13:34:29.514780 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:34:29.514749 2557 patch_prober.go:28] interesting pod/e2e-mt-7f8babf4-openshift-default-74f9755b6d-dzdc9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.61:15021/healthz/ready\": dial tcp 10.134.0.61:15021: connect: connection refused" start-of-body= Sep 10 13:34:29.514836 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:34:29.514795 2557 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-7f8babf4-openshift-default-74f9755b6d-dzdc9" podUID="287603f3-efd3-46a9-bbfe-312f96974f76" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.61:15021/healthz/ready\": dial tcp 10.134.0.61:15021: connect: connection refused" Sep 10 13:34:30.514496 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:34:30.514467 2557 patch_prober.go:28] interesting pod/e2e-mt-7f8babf4-openshift-default-74f9755b6d-dzdc9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.61:15021/healthz/ready\": dial tcp 10.134.0.61:15021: connect: connection refused" start-of-body= Sep 10 13:34:30.514865 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:34:30.514518 2557 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-7f8babf4-openshift-default-74f9755b6d-dzdc9" podUID="287603f3-efd3-46a9-bbfe-312f96974f76" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.61:15021/healthz/ready\": dial tcp 10.134.0.61:15021: connect: connection refused" Sep 10 13:34:31.515109 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:34:31.515075 2557 patch_prober.go:28] interesting pod/e2e-mt-7f8babf4-openshift-default-74f9755b6d-dzdc9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.61:15021/healthz/ready\": dial tcp 10.134.0.61:15021: connect: connection refused" start-of-body= Sep 10 13:34:31.515530 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:34:31.515140 2557 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-7f8babf4-openshift-default-74f9755b6d-dzdc9" podUID="287603f3-efd3-46a9-bbfe-312f96974f76" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.61:15021/healthz/ready\": dial tcp 10.134.0.61:15021: connect: connection refused" Sep 10 13:34:32.515340 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:34:32.515302 2557 patch_prober.go:28] interesting pod/e2e-mt-7f8babf4-openshift-default-74f9755b6d-dzdc9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.61:15021/healthz/ready\": dial tcp 10.134.0.61:15021: connect: connection refused" start-of-body= Sep 10 13:34:32.515847 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:34:32.515364 2557 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-7f8babf4-openshift-default-74f9755b6d-dzdc9" podUID="287603f3-efd3-46a9-bbfe-312f96974f76" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.61:15021/healthz/ready\": dial tcp 10.134.0.61:15021: connect: connection refused" Sep 10 13:34:33.514891 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:34:33.514861 2557 patch_prober.go:28] interesting pod/e2e-mt-7f8babf4-openshift-default-74f9755b6d-dzdc9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.61:15021/healthz/ready\": dial tcp 10.134.0.61:15021: connect: connection refused" start-of-body= Sep 10 13:34:33.515214 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:34:33.514920 2557 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-7f8babf4-openshift-default-74f9755b6d-dzdc9" podUID="287603f3-efd3-46a9-bbfe-312f96974f76" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.61:15021/healthz/ready\": dial tcp 10.134.0.61:15021: connect: connection refused" Sep 10 13:34:34.518688 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:34:34.518661 2557 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-7f8babf4-openshift-default-74f9755b6d-dzdc9" Sep 10 13:34:34.519066 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:34:34.518920 2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-7f8babf4-openshift-default-74f9755b6d-dzdc9" Sep 10 13:34:34.519607 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:34:34.519583 2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-7f8babf4-openshift-default-74f9755b6d-dzdc9" Sep 10 13:34:37.759505 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:34:37.759476 2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-6beee4-openshift-default-66d46f8786-2vth6"] Sep 10 13:34:37.759904 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:34:37.759720 2557 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-b-6beee4-openshift-default-66d46f8786-2vth6" podUID="b23f2560-5aa1-43b0-b0d9-674c6b0eb707" containerName="istio-proxy" containerID="cri-o://9a288f766cf74d1e24c89ffb2db04b173dc09e04adb328b49e8050ffe16f4b38" gracePeriod=30 Sep 10 13:34:38.746074 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:34:38.746038 2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w3-7657ef-openshift-default-85d55f76f6-gw88p"] Sep 10 13:34:38.746521 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:34:38.746504 2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bbab6794-a024-4b72-a8f2-2b9a4e85a3f2" containerName="istio-proxy" Sep 10 13:34:38.746628 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:34:38.746524 2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="bbab6794-a024-4b72-a8f2-2b9a4e85a3f2" containerName="istio-proxy" Sep 10 13:34:38.746628 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:34:38.746554 2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bbab6794-a024-4b72-a8f2-2b9a4e85a3f2" containerName="istio-proxy" Sep 10 13:34:38.746628 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:34:38.746563 2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="bbab6794-a024-4b72-a8f2-2b9a4e85a3f2" containerName="istio-proxy" Sep 10 13:34:38.746783 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:34:38.746677 2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="bbab6794-a024-4b72-a8f2-2b9a4e85a3f2" containerName="istio-proxy" Sep 10 13:34:38.749131 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:34:38.749112 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w3-7657ef-openshift-default-85d55f76f6-gw88p" Sep 10 13:34:38.750992 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:34:38.750967 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w3-7657ef-openshift-default-dockercfg-rkd6s\"" Sep 10 13:34:38.770616 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:34:38.770593 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-7657ef-openshift-default-85d55f76f6-gw88p"] Sep 10 13:34:38.853990 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:34:38.853950 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/979718af-bb04-448e-8c47-4c82a9adce63-istio-podinfo\") pod \"e2e-shared-a-w3-7657ef-openshift-default-85d55f76f6-gw88p\" (UID: \"979718af-bb04-448e-8c47-4c82a9adce63\") " pod="openshift-ingress/e2e-shared-a-w3-7657ef-openshift-default-85d55f76f6-gw88p" Sep 10 13:34:38.854189 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:34:38.854004 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/979718af-bb04-448e-8c47-4c82a9adce63-istio-data\") pod \"e2e-shared-a-w3-7657ef-openshift-default-85d55f76f6-gw88p\" (UID: \"979718af-bb04-448e-8c47-4c82a9adce63\") " pod="openshift-ingress/e2e-shared-a-w3-7657ef-openshift-default-85d55f76f6-gw88p" Sep 10 13:34:38.854189 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:34:38.854037 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/979718af-bb04-448e-8c47-4c82a9adce63-workload-socket\") pod \"e2e-shared-a-w3-7657ef-openshift-default-85d55f76f6-gw88p\" (UID: \"979718af-bb04-448e-8c47-4c82a9adce63\") " pod="openshift-ingress/e2e-shared-a-w3-7657ef-openshift-default-85d55f76f6-gw88p" Sep 10 13:34:38.854189 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:34:38.854064 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/979718af-bb04-448e-8c47-4c82a9adce63-credential-socket\") pod \"e2e-shared-a-w3-7657ef-openshift-default-85d55f76f6-gw88p\" (UID: \"979718af-bb04-448e-8c47-4c82a9adce63\") " pod="openshift-ingress/e2e-shared-a-w3-7657ef-openshift-default-85d55f76f6-gw88p" Sep 10 13:34:38.854189 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:34:38.854096 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/979718af-bb04-448e-8c47-4c82a9adce63-istio-envoy\") pod \"e2e-shared-a-w3-7657ef-openshift-default-85d55f76f6-gw88p\" (UID: \"979718af-bb04-448e-8c47-4c82a9adce63\") " pod="openshift-ingress/e2e-shared-a-w3-7657ef-openshift-default-85d55f76f6-gw88p" Sep 10 13:34:38.854414 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:34:38.854189 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/979718af-bb04-448e-8c47-4c82a9adce63-workload-certs\") pod \"e2e-shared-a-w3-7657ef-openshift-default-85d55f76f6-gw88p\" (UID: \"979718af-bb04-448e-8c47-4c82a9adce63\") " pod="openshift-ingress/e2e-shared-a-w3-7657ef-openshift-default-85d55f76f6-gw88p" Sep 10 13:34:38.854414 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:34:38.854262 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6k2jp\" (UniqueName: \"kubernetes.io/projected/979718af-bb04-448e-8c47-4c82a9adce63-kube-api-access-6k2jp\") pod \"e2e-shared-a-w3-7657ef-openshift-default-85d55f76f6-gw88p\" (UID: \"979718af-bb04-448e-8c47-4c82a9adce63\") " pod="openshift-ingress/e2e-shared-a-w3-7657ef-openshift-default-85d55f76f6-gw88p" Sep 10 13:34:38.854414 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:34:38.854316 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/979718af-bb04-448e-8c47-4c82a9adce63-istiod-ca-cert\") pod \"e2e-shared-a-w3-7657ef-openshift-default-85d55f76f6-gw88p\" (UID: \"979718af-bb04-448e-8c47-4c82a9adce63\") " pod="openshift-ingress/e2e-shared-a-w3-7657ef-openshift-default-85d55f76f6-gw88p" Sep 10 13:34:38.854414 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:34:38.854340 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/979718af-bb04-448e-8c47-4c82a9adce63-istio-token\") pod \"e2e-shared-a-w3-7657ef-openshift-default-85d55f76f6-gw88p\" (UID: \"979718af-bb04-448e-8c47-4c82a9adce63\") " pod="openshift-ingress/e2e-shared-a-w3-7657ef-openshift-default-85d55f76f6-gw88p" Sep 10 13:34:38.955399 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:34:38.955363 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/979718af-bb04-448e-8c47-4c82a9adce63-istio-envoy\") pod \"e2e-shared-a-w3-7657ef-openshift-default-85d55f76f6-gw88p\" (UID: \"979718af-bb04-448e-8c47-4c82a9adce63\") " pod="openshift-ingress/e2e-shared-a-w3-7657ef-openshift-default-85d55f76f6-gw88p" Sep 10 13:34:38.955608 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:34:38.955466 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/979718af-bb04-448e-8c47-4c82a9adce63-workload-certs\") pod \"e2e-shared-a-w3-7657ef-openshift-default-85d55f76f6-gw88p\" (UID: \"979718af-bb04-448e-8c47-4c82a9adce63\") " pod="openshift-ingress/e2e-shared-a-w3-7657ef-openshift-default-85d55f76f6-gw88p" Sep 10 13:34:38.955608 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:34:38.955534 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6k2jp\" (UniqueName: \"kubernetes.io/projected/979718af-bb04-448e-8c47-4c82a9adce63-kube-api-access-6k2jp\") pod \"e2e-shared-a-w3-7657ef-openshift-default-85d55f76f6-gw88p\" (UID: \"979718af-bb04-448e-8c47-4c82a9adce63\") " pod="openshift-ingress/e2e-shared-a-w3-7657ef-openshift-default-85d55f76f6-gw88p" Sep 10 13:34:38.955608 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:34:38.955587 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/979718af-bb04-448e-8c47-4c82a9adce63-istiod-ca-cert\") pod \"e2e-shared-a-w3-7657ef-openshift-default-85d55f76f6-gw88p\" (UID: \"979718af-bb04-448e-8c47-4c82a9adce63\") " pod="openshift-ingress/e2e-shared-a-w3-7657ef-openshift-default-85d55f76f6-gw88p" Sep 10 13:34:38.955778 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:34:38.955613 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/979718af-bb04-448e-8c47-4c82a9adce63-istio-token\") pod \"e2e-shared-a-w3-7657ef-openshift-default-85d55f76f6-gw88p\" (UID: \"979718af-bb04-448e-8c47-4c82a9adce63\") " pod="openshift-ingress/e2e-shared-a-w3-7657ef-openshift-default-85d55f76f6-gw88p" Sep 10 13:34:38.955778 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:34:38.955658 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/979718af-bb04-448e-8c47-4c82a9adce63-istio-podinfo\") pod \"e2e-shared-a-w3-7657ef-openshift-default-85d55f76f6-gw88p\" (UID: \"979718af-bb04-448e-8c47-4c82a9adce63\") " pod="openshift-ingress/e2e-shared-a-w3-7657ef-openshift-default-85d55f76f6-gw88p" Sep 10 13:34:38.955778 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:34:38.955689 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/979718af-bb04-448e-8c47-4c82a9adce63-istio-data\") pod \"e2e-shared-a-w3-7657ef-openshift-default-85d55f76f6-gw88p\" (UID: \"979718af-bb04-448e-8c47-4c82a9adce63\") " pod="openshift-ingress/e2e-shared-a-w3-7657ef-openshift-default-85d55f76f6-gw88p" Sep 10 13:34:38.955778 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:34:38.955715 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/979718af-bb04-448e-8c47-4c82a9adce63-workload-socket\") pod \"e2e-shared-a-w3-7657ef-openshift-default-85d55f76f6-gw88p\" (UID: \"979718af-bb04-448e-8c47-4c82a9adce63\") " pod="openshift-ingress/e2e-shared-a-w3-7657ef-openshift-default-85d55f76f6-gw88p" Sep 10 13:34:38.955778 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:34:38.955738 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/979718af-bb04-448e-8c47-4c82a9adce63-credential-socket\") pod \"e2e-shared-a-w3-7657ef-openshift-default-85d55f76f6-gw88p\" (UID: \"979718af-bb04-448e-8c47-4c82a9adce63\") " pod="openshift-ingress/e2e-shared-a-w3-7657ef-openshift-default-85d55f76f6-gw88p" Sep 10 13:34:38.956036 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:34:38.955931 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/979718af-bb04-448e-8c47-4c82a9adce63-workload-certs\") pod \"e2e-shared-a-w3-7657ef-openshift-default-85d55f76f6-gw88p\" (UID: \"979718af-bb04-448e-8c47-4c82a9adce63\") " pod="openshift-ingress/e2e-shared-a-w3-7657ef-openshift-default-85d55f76f6-gw88p" Sep 10 13:34:38.956115 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:34:38.956091 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/979718af-bb04-448e-8c47-4c82a9adce63-istio-data\") pod \"e2e-shared-a-w3-7657ef-openshift-default-85d55f76f6-gw88p\" (UID: \"979718af-bb04-448e-8c47-4c82a9adce63\") " pod="openshift-ingress/e2e-shared-a-w3-7657ef-openshift-default-85d55f76f6-gw88p" Sep 10 13:34:38.956164 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:34:38.956146 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/979718af-bb04-448e-8c47-4c82a9adce63-credential-socket\") pod \"e2e-shared-a-w3-7657ef-openshift-default-85d55f76f6-gw88p\" (UID: \"979718af-bb04-448e-8c47-4c82a9adce63\") " pod="openshift-ingress/e2e-shared-a-w3-7657ef-openshift-default-85d55f76f6-gw88p" Sep 10 13:34:38.956368 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:34:38.956354 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/979718af-bb04-448e-8c47-4c82a9adce63-workload-socket\") pod \"e2e-shared-a-w3-7657ef-openshift-default-85d55f76f6-gw88p\" (UID: \"979718af-bb04-448e-8c47-4c82a9adce63\") " pod="openshift-ingress/e2e-shared-a-w3-7657ef-openshift-default-85d55f76f6-gw88p" Sep 10 13:34:38.956643 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:34:38.956619 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/979718af-bb04-448e-8c47-4c82a9adce63-istiod-ca-cert\") pod \"e2e-shared-a-w3-7657ef-openshift-default-85d55f76f6-gw88p\" (UID: \"979718af-bb04-448e-8c47-4c82a9adce63\") " pod="openshift-ingress/e2e-shared-a-w3-7657ef-openshift-default-85d55f76f6-gw88p" Sep 10 13:34:38.957900 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:34:38.957884 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/979718af-bb04-448e-8c47-4c82a9adce63-istio-envoy\") pod \"e2e-shared-a-w3-7657ef-openshift-default-85d55f76f6-gw88p\" (UID: \"979718af-bb04-448e-8c47-4c82a9adce63\") " pod="openshift-ingress/e2e-shared-a-w3-7657ef-openshift-default-85d55f76f6-gw88p" Sep 10 13:34:38.958112 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:34:38.958094 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/979718af-bb04-448e-8c47-4c82a9adce63-istio-podinfo\") pod \"e2e-shared-a-w3-7657ef-openshift-default-85d55f76f6-gw88p\" (UID: \"979718af-bb04-448e-8c47-4c82a9adce63\") " pod="openshift-ingress/e2e-shared-a-w3-7657ef-openshift-default-85d55f76f6-gw88p" Sep 10 13:34:38.987929 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:34:38.987906 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6k2jp\" (UniqueName: \"kubernetes.io/projected/979718af-bb04-448e-8c47-4c82a9adce63-kube-api-access-6k2jp\") pod \"e2e-shared-a-w3-7657ef-openshift-default-85d55f76f6-gw88p\" (UID: \"979718af-bb04-448e-8c47-4c82a9adce63\") " pod="openshift-ingress/e2e-shared-a-w3-7657ef-openshift-default-85d55f76f6-gw88p" Sep 10 13:34:38.997663 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:34:38.997591 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/979718af-bb04-448e-8c47-4c82a9adce63-istio-token\") pod \"e2e-shared-a-w3-7657ef-openshift-default-85d55f76f6-gw88p\" (UID: \"979718af-bb04-448e-8c47-4c82a9adce63\") " pod="openshift-ingress/e2e-shared-a-w3-7657ef-openshift-default-85d55f76f6-gw88p" Sep 10 13:34:39.059828 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:34:39.059794 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w3-7657ef-openshift-default-85d55f76f6-gw88p" Sep 10 13:34:39.340893 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:34:39.340869 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-7657ef-openshift-default-85d55f76f6-gw88p"] Sep 10 13:34:39.342287 ip-10-0-139-174 kubenswrapper[2557]: W0910 13:34:39.342261 2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod979718af_bb04_448e_8c47_4c82a9adce63.slice/crio-34ca8ce9cde3717101e20496d77e051311fa6e34f2de5651fc38815ac6c8a63a WatchSource:0}: Error finding container 34ca8ce9cde3717101e20496d77e051311fa6e34f2de5651fc38815ac6c8a63a: Status 404 returned error can't find the container with id 34ca8ce9cde3717101e20496d77e051311fa6e34f2de5651fc38815ac6c8a63a Sep 10 13:34:39.344468 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:34:39.344440 2557 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892152Ki","pods":"250"} Sep 10 13:34:39.344538 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:34:39.344506 2557 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892152Ki","pods":"250"} Sep 10 13:34:39.344576 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:34:39.344555 2557 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892152Ki","pods":"250"} Sep 10 13:34:40.152055 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:34:40.152012 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-7657ef-openshift-default-85d55f76f6-gw88p" event={"ID":"979718af-bb04-448e-8c47-4c82a9adce63","Type":"ContainerStarted","Data":"c9bc7d0ceed89ce9f9cc0ad949f01ea822c607e00482217d447fb53735bfdb0e"} Sep 10 13:34:40.152474 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:34:40.152099 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-7657ef-openshift-default-85d55f76f6-gw88p" event={"ID":"979718af-bb04-448e-8c47-4c82a9adce63","Type":"ContainerStarted","Data":"34ca8ce9cde3717101e20496d77e051311fa6e34f2de5651fc38815ac6c8a63a"} Sep 10 13:34:40.194063 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:34:40.194014 2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w3-7657ef-openshift-default-85d55f76f6-gw88p" podStartSLOduration=2.193999579 podStartE2EDuration="2.193999579s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 13:34:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 13:34:40.192836718 +0000 UTC m=+1040.499831131" watchObservedRunningTime="2026-09-10 13:34:40.193999579 +0000 UTC m=+1040.500993993" Sep 10 13:34:41.060343 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:34:41.060306 2557 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w3-7657ef-openshift-default-85d55f76f6-gw88p" Sep 10 13:34:41.065151 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:34:41.065124 2557 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w3-7657ef-openshift-default-85d55f76f6-gw88p" Sep 10 13:34:41.158293 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:34:41.158259 2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w3-7657ef-openshift-default-85d55f76f6-gw88p" Sep 10 13:34:41.159143 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:34:41.159122 2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w3-7657ef-openshift-default-85d55f76f6-gw88p" Sep 10 13:34:43.005810 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:34:43.005790 2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-6beee4-openshift-default-66d46f8786-2vth6" Sep 10 13:34:43.102308 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:34:43.102239 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b23f2560-5aa1-43b0-b0d9-674c6b0eb707-istio-envoy\") pod \"b23f2560-5aa1-43b0-b0d9-674c6b0eb707\" (UID: \"b23f2560-5aa1-43b0-b0d9-674c6b0eb707\") " Sep 10 13:34:43.102308 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:34:43.102299 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b23f2560-5aa1-43b0-b0d9-674c6b0eb707-workload-socket\") pod \"b23f2560-5aa1-43b0-b0d9-674c6b0eb707\" (UID: \"b23f2560-5aa1-43b0-b0d9-674c6b0eb707\") " Sep 10 13:34:43.102526 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:34:43.102383 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b23f2560-5aa1-43b0-b0d9-674c6b0eb707-istio-data\") pod \"b23f2560-5aa1-43b0-b0d9-674c6b0eb707\" (UID: \"b23f2560-5aa1-43b0-b0d9-674c6b0eb707\") " Sep 10 13:34:43.102526 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:34:43.102399 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b23f2560-5aa1-43b0-b0d9-674c6b0eb707-istiod-ca-cert\") pod \"b23f2560-5aa1-43b0-b0d9-674c6b0eb707\" (UID: \"b23f2560-5aa1-43b0-b0d9-674c6b0eb707\") " Sep 10 13:34:43.102526 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:34:43.102441 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b23f2560-5aa1-43b0-b0d9-674c6b0eb707-istio-podinfo\") pod \"b23f2560-5aa1-43b0-b0d9-674c6b0eb707\" (UID: \"b23f2560-5aa1-43b0-b0d9-674c6b0eb707\") " Sep 10 13:34:43.102526 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:34:43.102479 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b23f2560-5aa1-43b0-b0d9-674c6b0eb707-workload-certs\") pod \"b23f2560-5aa1-43b0-b0d9-674c6b0eb707\" (UID: \"b23f2560-5aa1-43b0-b0d9-674c6b0eb707\") " Sep 10 13:34:43.102526 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:34:43.102503 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b23f2560-5aa1-43b0-b0d9-674c6b0eb707-credential-socket\") pod \"b23f2560-5aa1-43b0-b0d9-674c6b0eb707\" (UID: \"b23f2560-5aa1-43b0-b0d9-674c6b0eb707\") " Sep 10 13:34:43.102790 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:34:43.102552 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b23f2560-5aa1-43b0-b0d9-674c6b0eb707-istio-token\") pod \"b23f2560-5aa1-43b0-b0d9-674c6b0eb707\" (UID: \"b23f2560-5aa1-43b0-b0d9-674c6b0eb707\") " Sep 10 13:34:43.102790 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:34:43.102586 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-htr6w\" (UniqueName: \"kubernetes.io/projected/b23f2560-5aa1-43b0-b0d9-674c6b0eb707-kube-api-access-htr6w\") pod \"b23f2560-5aa1-43b0-b0d9-674c6b0eb707\" (UID: \"b23f2560-5aa1-43b0-b0d9-674c6b0eb707\") " Sep 10 13:34:43.102790 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:34:43.102602 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b23f2560-5aa1-43b0-b0d9-674c6b0eb707-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "b23f2560-5aa1-43b0-b0d9-674c6b0eb707" (UID: "b23f2560-5aa1-43b0-b0d9-674c6b0eb707"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 13:34:43.102790 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:34:43.102732 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b23f2560-5aa1-43b0-b0d9-674c6b0eb707-istio-data" (OuterVolumeSpecName: "istio-data") pod "b23f2560-5aa1-43b0-b0d9-674c6b0eb707" (UID: "b23f2560-5aa1-43b0-b0d9-674c6b0eb707"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 13:34:43.103002 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:34:43.102792 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b23f2560-5aa1-43b0-b0d9-674c6b0eb707-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "b23f2560-5aa1-43b0-b0d9-674c6b0eb707" (UID: "b23f2560-5aa1-43b0-b0d9-674c6b0eb707"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 13:34:43.103002 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:34:43.102823 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b23f2560-5aa1-43b0-b0d9-674c6b0eb707-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "b23f2560-5aa1-43b0-b0d9-674c6b0eb707" (UID: "b23f2560-5aa1-43b0-b0d9-674c6b0eb707"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 10 13:34:43.103002 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:34:43.102912 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b23f2560-5aa1-43b0-b0d9-674c6b0eb707-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "b23f2560-5aa1-43b0-b0d9-674c6b0eb707" (UID: "b23f2560-5aa1-43b0-b0d9-674c6b0eb707"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 13:34:43.103165 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:34:43.103135 2557 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b23f2560-5aa1-43b0-b0d9-674c6b0eb707-workload-socket\") on node \"ip-10-0-139-174.ec2.internal\" DevicePath \"\"" Sep 10 13:34:43.103165 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:34:43.103156 2557 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b23f2560-5aa1-43b0-b0d9-674c6b0eb707-istio-data\") on node \"ip-10-0-139-174.ec2.internal\" DevicePath \"\"" Sep 10 13:34:43.103459 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:34:43.103171 2557 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b23f2560-5aa1-43b0-b0d9-674c6b0eb707-istiod-ca-cert\") on node \"ip-10-0-139-174.ec2.internal\" DevicePath \"\"" Sep 10 13:34:43.103459 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:34:43.103185 2557 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b23f2560-5aa1-43b0-b0d9-674c6b0eb707-workload-certs\") on node \"ip-10-0-139-174.ec2.internal\" DevicePath \"\"" Sep 10 13:34:43.103459 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:34:43.103199 2557 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b23f2560-5aa1-43b0-b0d9-674c6b0eb707-credential-socket\") on node \"ip-10-0-139-174.ec2.internal\" DevicePath \"\"" Sep 10 13:34:43.104562 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:34:43.104537 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b23f2560-5aa1-43b0-b0d9-674c6b0eb707-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "b23f2560-5aa1-43b0-b0d9-674c6b0eb707" (UID: "b23f2560-5aa1-43b0-b0d9-674c6b0eb707"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 13:34:43.104861 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:34:43.104834 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b23f2560-5aa1-43b0-b0d9-674c6b0eb707-istio-token" (OuterVolumeSpecName: "istio-token") pod "b23f2560-5aa1-43b0-b0d9-674c6b0eb707" (UID: "b23f2560-5aa1-43b0-b0d9-674c6b0eb707"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 10 13:34:43.104934 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:34:43.104839 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b23f2560-5aa1-43b0-b0d9-674c6b0eb707-kube-api-access-htr6w" (OuterVolumeSpecName: "kube-api-access-htr6w") pod "b23f2560-5aa1-43b0-b0d9-674c6b0eb707" (UID: "b23f2560-5aa1-43b0-b0d9-674c6b0eb707"). InnerVolumeSpecName "kube-api-access-htr6w". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 10 13:34:43.104934 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:34:43.104866 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/b23f2560-5aa1-43b0-b0d9-674c6b0eb707-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "b23f2560-5aa1-43b0-b0d9-674c6b0eb707" (UID: "b23f2560-5aa1-43b0-b0d9-674c6b0eb707"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Sep 10 13:34:43.171849 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:34:43.171824 2557 generic.go:356] "Generic (PLEG): container finished" podID="b23f2560-5aa1-43b0-b0d9-674c6b0eb707" containerID="9a288f766cf74d1e24c89ffb2db04b173dc09e04adb328b49e8050ffe16f4b38" exitCode=0 Sep 10 13:34:43.171967 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:34:43.171881 2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-6beee4-openshift-default-66d46f8786-2vth6" Sep 10 13:34:43.171967 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:34:43.171913 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-6beee4-openshift-default-66d46f8786-2vth6" event={"ID":"b23f2560-5aa1-43b0-b0d9-674c6b0eb707","Type":"ContainerDied","Data":"9a288f766cf74d1e24c89ffb2db04b173dc09e04adb328b49e8050ffe16f4b38"} Sep 10 13:34:43.172082 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:34:43.171971 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-6beee4-openshift-default-66d46f8786-2vth6" event={"ID":"b23f2560-5aa1-43b0-b0d9-674c6b0eb707","Type":"ContainerDied","Data":"b03ee78ce8d64fb64faa2c8e36a213bf288ffcd00adfae9bb6d813f400108278"} Sep 10 13:34:43.172082 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:34:43.171995 2557 scope.go:117] "RemoveContainer" containerID="9a288f766cf74d1e24c89ffb2db04b173dc09e04adb328b49e8050ffe16f4b38" Sep 10 13:34:43.181643 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:34:43.181628 2557 scope.go:117] "RemoveContainer" containerID="843b18ac4f0ed976f156adbefa14a0a84d2086a03424b3842baa5ebbfdbb47a5" Sep 10 13:34:43.190353 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:34:43.190338 2557 scope.go:117] "RemoveContainer" containerID="9a288f766cf74d1e24c89ffb2db04b173dc09e04adb328b49e8050ffe16f4b38" Sep 10 13:34:43.190608 ip-10-0-139-174 kubenswrapper[2557]: E0910 13:34:43.190592 2557 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9a288f766cf74d1e24c89ffb2db04b173dc09e04adb328b49e8050ffe16f4b38\": container with ID starting with 9a288f766cf74d1e24c89ffb2db04b173dc09e04adb328b49e8050ffe16f4b38 not found: ID does not exist" containerID="9a288f766cf74d1e24c89ffb2db04b173dc09e04adb328b49e8050ffe16f4b38" Sep 10 13:34:43.190660 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:34:43.190614 2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9a288f766cf74d1e24c89ffb2db04b173dc09e04adb328b49e8050ffe16f4b38"} err="failed to get container status \"9a288f766cf74d1e24c89ffb2db04b173dc09e04adb328b49e8050ffe16f4b38\": rpc error: code = NotFound desc = could not find container \"9a288f766cf74d1e24c89ffb2db04b173dc09e04adb328b49e8050ffe16f4b38\": container with ID starting with 9a288f766cf74d1e24c89ffb2db04b173dc09e04adb328b49e8050ffe16f4b38 not found: ID does not exist" Sep 10 13:34:43.190660 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:34:43.190629 2557 scope.go:117] "RemoveContainer" containerID="843b18ac4f0ed976f156adbefa14a0a84d2086a03424b3842baa5ebbfdbb47a5" Sep 10 13:34:43.190861 ip-10-0-139-174 kubenswrapper[2557]: E0910 13:34:43.190843 2557 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"843b18ac4f0ed976f156adbefa14a0a84d2086a03424b3842baa5ebbfdbb47a5\": container with ID starting with 843b18ac4f0ed976f156adbefa14a0a84d2086a03424b3842baa5ebbfdbb47a5 not found: ID does not exist" containerID="843b18ac4f0ed976f156adbefa14a0a84d2086a03424b3842baa5ebbfdbb47a5" Sep 10 13:34:43.190918 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:34:43.190872 2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"843b18ac4f0ed976f156adbefa14a0a84d2086a03424b3842baa5ebbfdbb47a5"} err="failed to get container status \"843b18ac4f0ed976f156adbefa14a0a84d2086a03424b3842baa5ebbfdbb47a5\": rpc error: code = NotFound desc = could not find container \"843b18ac4f0ed976f156adbefa14a0a84d2086a03424b3842baa5ebbfdbb47a5\": container with ID starting with 843b18ac4f0ed976f156adbefa14a0a84d2086a03424b3842baa5ebbfdbb47a5 not found: ID does not exist" Sep 10 13:34:43.198926 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:34:43.198899 2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-6beee4-openshift-default-66d46f8786-2vth6"] Sep 10 13:34:43.203376 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:34:43.203355 2557 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-b-6beee4-openshift-default-66d46f8786-2vth6"] Sep 10 13:34:43.203639 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:34:43.203621 2557 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b23f2560-5aa1-43b0-b0d9-674c6b0eb707-istio-podinfo\") on node \"ip-10-0-139-174.ec2.internal\" DevicePath \"\"" Sep 10 13:34:43.203685 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:34:43.203643 2557 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b23f2560-5aa1-43b0-b0d9-674c6b0eb707-istio-token\") on node \"ip-10-0-139-174.ec2.internal\" DevicePath \"\"" Sep 10 13:34:43.203685 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:34:43.203652 2557 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-htr6w\" (UniqueName: \"kubernetes.io/projected/b23f2560-5aa1-43b0-b0d9-674c6b0eb707-kube-api-access-htr6w\") on node \"ip-10-0-139-174.ec2.internal\" DevicePath \"\"" Sep 10 13:34:43.203685 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:34:43.203662 2557 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b23f2560-5aa1-43b0-b0d9-674c6b0eb707-istio-envoy\") on node \"ip-10-0-139-174.ec2.internal\" DevicePath \"\"" Sep 10 13:34:44.220348 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:34:44.220310 2557 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b23f2560-5aa1-43b0-b0d9-674c6b0eb707" path="/var/lib/kubelet/pods/b23f2560-5aa1-43b0-b0d9-674c6b0eb707/volumes" Sep 10 13:34:46.789536 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:34:46.789507 2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-7f8babf4-498mk"] Sep 10 13:34:46.790094 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:34:46.790074 2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b23f2560-5aa1-43b0-b0d9-674c6b0eb707" containerName="istio-proxy" Sep 10 13:34:46.790185 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:34:46.790097 2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="b23f2560-5aa1-43b0-b0d9-674c6b0eb707" containerName="istio-proxy" Sep 10 13:34:46.790244 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:34:46.790223 2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="b23f2560-5aa1-43b0-b0d9-674c6b0eb707" containerName="istio-proxy" Sep 10 13:34:46.790244 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:34:46.790235 2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="bbab6794-a024-4b72-a8f2-2b9a4e85a3f2" containerName="istio-proxy" Sep 10 13:34:46.790348 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:34:46.790246 2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="b23f2560-5aa1-43b0-b0d9-674c6b0eb707" containerName="istio-proxy" Sep 10 13:34:46.793350 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:34:46.793329 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-7f8babf4-498mk" Sep 10 13:34:46.795672 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:34:46.795648 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-wtblv\"" Sep 10 13:34:46.795774 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:34:46.795651 2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Sep 10 13:34:46.802638 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:34:46.802617 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-7f8babf4-498mk"] Sep 10 13:34:46.842918 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:34:46.842887 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/bd8e0083-3d60-4b95-9ca5-9b54acb813df-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-7f8babf4-498mk\" (UID: \"bd8e0083-3d60-4b95-9ca5-9b54acb813df\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-7f8babf4-498mk" Sep 10 13:34:46.944324 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:34:46.944294 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/bd8e0083-3d60-4b95-9ca5-9b54acb813df-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-7f8babf4-498mk\" (UID: \"bd8e0083-3d60-4b95-9ca5-9b54acb813df\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-7f8babf4-498mk" Sep 10 13:34:46.945000 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:34:46.944970 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/bd8e0083-3d60-4b95-9ca5-9b54acb813df-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-7f8babf4-498mk\" (UID: \"bd8e0083-3d60-4b95-9ca5-9b54acb813df\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-7f8babf4-498mk" Sep 10 13:34:47.105784 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:34:47.105698 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-7f8babf4-498mk" Sep 10 13:34:47.392338 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:34:47.392313 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-7f8babf4-498mk"] Sep 10 13:34:47.394239 ip-10-0-139-174 kubenswrapper[2557]: W0910 13:34:47.394214 2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbd8e0083_3d60_4b95_9ca5_9b54acb813df.slice/crio-e82e430b7c18d731aa84b140fa9596b00e5860712568470b5e592cac63d373b0 WatchSource:0}: Error finding container e82e430b7c18d731aa84b140fa9596b00e5860712568470b5e592cac63d373b0: Status 404 returned error can't find the container with id e82e430b7c18d731aa84b140fa9596b00e5860712568470b5e592cac63d373b0 Sep 10 13:34:48.203608 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:34:48.203577 2557 generic.go:356] "Generic (PLEG): container finished" podID="bd8e0083-3d60-4b95-9ca5-9b54acb813df" containerID="d8475533d299565f50768b063a38304850578f83f458a75ff3d4eedca28a2fbe" exitCode=0 Sep 10 13:34:48.203969 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:34:48.203662 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-7f8babf4-498mk" event={"ID":"bd8e0083-3d60-4b95-9ca5-9b54acb813df","Type":"ContainerDied","Data":"d8475533d299565f50768b063a38304850578f83f458a75ff3d4eedca28a2fbe"} Sep 10 13:34:48.203969 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:34:48.203717 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-7f8babf4-498mk" event={"ID":"bd8e0083-3d60-4b95-9ca5-9b54acb813df","Type":"ContainerStarted","Data":"e82e430b7c18d731aa84b140fa9596b00e5860712568470b5e592cac63d373b0"} Sep 10 13:34:49.344055 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:34:49.344035 2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-7f8babf4-498mk" Sep 10 13:34:49.367196 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:34:49.367149 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/bd8e0083-3d60-4b95-9ca5-9b54acb813df-maas-api-service-ca\") pod \"bd8e0083-3d60-4b95-9ca5-9b54acb813df\" (UID: \"bd8e0083-3d60-4b95-9ca5-9b54acb813df\") " Sep 10 13:34:49.367771 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:34:49.367743 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bd8e0083-3d60-4b95-9ca5-9b54acb813df-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "bd8e0083-3d60-4b95-9ca5-9b54acb813df" (UID: "bd8e0083-3d60-4b95-9ca5-9b54acb813df"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 10 13:34:49.469201 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:34:49.469055 2557 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/bd8e0083-3d60-4b95-9ca5-9b54acb813df-maas-api-service-ca\") on node \"ip-10-0-139-174.ec2.internal\" DevicePath \"\"" Sep 10 13:34:50.224008 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:34:50.223965 2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-7f8babf4-498mk" Sep 10 13:34:50.226889 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:34:50.226860 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-7f8babf4-498mk" event={"ID":"bd8e0083-3d60-4b95-9ca5-9b54acb813df","Type":"ContainerDied","Data":"e82e430b7c18d731aa84b140fa9596b00e5860712568470b5e592cac63d373b0"} Sep 10 13:34:50.227048 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:34:50.226897 2557 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e82e430b7c18d731aa84b140fa9596b00e5860712568470b5e592cac63d373b0" Sep 10 13:34:57.199589 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:34:57.199550 2557 patch_prober.go:28] interesting pod/e2e-shared-a-w3-7657ef-openshift-default-85d55f76f6-gw88p container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.63:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Sep 10 13:34:57.199994 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:34:57.199617 2557 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-a-w3-7657ef-openshift-default-85d55f76f6-gw88p" podUID="979718af-bb04-448e-8c47-4c82a9adce63" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.63:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Sep 10 13:34:57.935263 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:34:57.935229 2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-7f8babf4-6c88b456b5-c8bpr"] Sep 10 13:34:57.935489 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:34:57.935468 2557 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-mt-7f8babf4-6c88b456b5-c8bpr" podUID="3da88b7f-bbfb-4521-acc5-dfe20cacbced" containerName="payload-processing" containerID="cri-o://fb83f56d7a4b17a9ea1ea0a3068fbe340d78d6d357beb3a8b4305b671b5e360e" gracePeriod=30 Sep 10 13:34:58.194778 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:34:58.194758 2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-7f8babf4-6c88b456b5-c8bpr" Sep 10 13:34:58.256786 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:34:58.256664 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-w9xxn\" (UniqueName: \"kubernetes.io/projected/3da88b7f-bbfb-4521-acc5-dfe20cacbced-kube-api-access-w9xxn\") pod \"3da88b7f-bbfb-4521-acc5-dfe20cacbced\" (UID: \"3da88b7f-bbfb-4521-acc5-dfe20cacbced\") " Sep 10 13:34:58.257560 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:34:58.256849 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3da88b7f-bbfb-4521-acc5-dfe20cacbced-plugins-config-volume\") pod \"3da88b7f-bbfb-4521-acc5-dfe20cacbced\" (UID: \"3da88b7f-bbfb-4521-acc5-dfe20cacbced\") " Sep 10 13:34:58.257953 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:34:58.257922 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3da88b7f-bbfb-4521-acc5-dfe20cacbced-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "3da88b7f-bbfb-4521-acc5-dfe20cacbced" (UID: "3da88b7f-bbfb-4521-acc5-dfe20cacbced"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 10 13:34:58.260950 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:34:58.260909 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3da88b7f-bbfb-4521-acc5-dfe20cacbced-kube-api-access-w9xxn" (OuterVolumeSpecName: "kube-api-access-w9xxn") pod "3da88b7f-bbfb-4521-acc5-dfe20cacbced" (UID: "3da88b7f-bbfb-4521-acc5-dfe20cacbced"). InnerVolumeSpecName "kube-api-access-w9xxn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 10 13:34:58.281000 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:34:58.280958 2557 generic.go:356] "Generic (PLEG): container finished" podID="3da88b7f-bbfb-4521-acc5-dfe20cacbced" containerID="fb83f56d7a4b17a9ea1ea0a3068fbe340d78d6d357beb3a8b4305b671b5e360e" exitCode=0 Sep 10 13:34:58.281185 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:34:58.281076 2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-7f8babf4-6c88b456b5-c8bpr" Sep 10 13:34:58.281185 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:34:58.281097 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-7f8babf4-6c88b456b5-c8bpr" event={"ID":"3da88b7f-bbfb-4521-acc5-dfe20cacbced","Type":"ContainerDied","Data":"fb83f56d7a4b17a9ea1ea0a3068fbe340d78d6d357beb3a8b4305b671b5e360e"} Sep 10 13:34:58.281185 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:34:58.281153 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-7f8babf4-6c88b456b5-c8bpr" event={"ID":"3da88b7f-bbfb-4521-acc5-dfe20cacbced","Type":"ContainerDied","Data":"15fa18060f16452a74548b5d0081bccd0af185f8e8a30257102850d804ff95aa"} Sep 10 13:34:58.281185 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:34:58.281184 2557 scope.go:117] "RemoveContainer" containerID="fb83f56d7a4b17a9ea1ea0a3068fbe340d78d6d357beb3a8b4305b671b5e360e" Sep 10 13:34:58.290582 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:34:58.289488 2557 generic.go:356] "Generic (PLEG): container finished" podID="979718af-bb04-448e-8c47-4c82a9adce63" containerID="c9bc7d0ceed89ce9f9cc0ad949f01ea822c607e00482217d447fb53735bfdb0e" exitCode=0 Sep 10 13:34:58.290582 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:34:58.289637 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-7657ef-openshift-default-85d55f76f6-gw88p" event={"ID":"979718af-bb04-448e-8c47-4c82a9adce63","Type":"ContainerDied","Data":"c9bc7d0ceed89ce9f9cc0ad949f01ea822c607e00482217d447fb53735bfdb0e"} Sep 10 13:34:58.290582 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:34:58.290324 2557 scope.go:117] "RemoveContainer" containerID="c9bc7d0ceed89ce9f9cc0ad949f01ea822c607e00482217d447fb53735bfdb0e" Sep 10 13:34:58.292809 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:34:58.292768 2557 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892152Ki","pods":"250"} Sep 10 13:34:58.292937 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:34:58.292912 2557 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892152Ki","pods":"250"} Sep 10 13:34:58.293559 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:34:58.293009 2557 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892152Ki","pods":"250"} Sep 10 13:34:58.305243 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:34:58.305215 2557 scope.go:117] "RemoveContainer" containerID="fb83f56d7a4b17a9ea1ea0a3068fbe340d78d6d357beb3a8b4305b671b5e360e" Sep 10 13:34:58.305626 ip-10-0-139-174 kubenswrapper[2557]: E0910 13:34:58.305600 2557 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fb83f56d7a4b17a9ea1ea0a3068fbe340d78d6d357beb3a8b4305b671b5e360e\": container with ID starting with fb83f56d7a4b17a9ea1ea0a3068fbe340d78d6d357beb3a8b4305b671b5e360e not found: ID does not exist" containerID="fb83f56d7a4b17a9ea1ea0a3068fbe340d78d6d357beb3a8b4305b671b5e360e" Sep 10 13:34:58.305718 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:34:58.305640 2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fb83f56d7a4b17a9ea1ea0a3068fbe340d78d6d357beb3a8b4305b671b5e360e"} err="failed to get container status \"fb83f56d7a4b17a9ea1ea0a3068fbe340d78d6d357beb3a8b4305b671b5e360e\": rpc error: code = NotFound desc = could not find container \"fb83f56d7a4b17a9ea1ea0a3068fbe340d78d6d357beb3a8b4305b671b5e360e\": container with ID starting with fb83f56d7a4b17a9ea1ea0a3068fbe340d78d6d357beb3a8b4305b671b5e360e not found: ID does not exist" Sep 10 13:34:58.359156 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:34:58.359128 2557 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w9xxn\" (UniqueName: \"kubernetes.io/projected/3da88b7f-bbfb-4521-acc5-dfe20cacbced-kube-api-access-w9xxn\") on node \"ip-10-0-139-174.ec2.internal\" DevicePath \"\"" Sep 10 13:34:58.359156 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:34:58.359153 2557 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3da88b7f-bbfb-4521-acc5-dfe20cacbced-plugins-config-volume\") on node \"ip-10-0-139-174.ec2.internal\" DevicePath \"\"" Sep 10 13:34:58.388948 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:34:58.388904 2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-7f8babf4-6c88b456b5-c8bpr"] Sep 10 13:34:58.396669 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:34:58.396635 2557 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-7f8babf4-6c88b456b5-c8bpr"] Sep 10 13:34:58.456243 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:34:58.456153 2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-7f8babf4-498mk"] Sep 10 13:34:58.464839 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:34:58.464753 2557 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-7f8babf4-498mk"] Sep 10 13:34:59.298487 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:34:59.298451 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-7657ef-openshift-default-85d55f76f6-gw88p" event={"ID":"979718af-bb04-448e-8c47-4c82a9adce63","Type":"ContainerStarted","Data":"ad663552ea2d18c5a57c99e5eabe50898a9deceaa2491c336b626268f58d9acb"} Sep 10 13:35:00.061098 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:35:00.061046 2557 patch_prober.go:28] interesting pod/e2e-shared-a-w3-7657ef-openshift-default-85d55f76f6-gw88p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.63:15021/healthz/ready\": dial tcp 10.134.0.63:15021: connect: connection refused" start-of-body= Sep 10 13:35:00.061333 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:35:00.061124 2557 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-7657ef-openshift-default-85d55f76f6-gw88p" podUID="979718af-bb04-448e-8c47-4c82a9adce63" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.63:15021/healthz/ready\": dial tcp 10.134.0.63:15021: connect: connection refused" Sep 10 13:35:00.061472 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:35:00.061448 2557 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w3-7657ef-openshift-default-85d55f76f6-gw88p" Sep 10 13:35:00.220708 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:35:00.220672 2557 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3da88b7f-bbfb-4521-acc5-dfe20cacbced" path="/var/lib/kubelet/pods/3da88b7f-bbfb-4521-acc5-dfe20cacbced/volumes" Sep 10 13:35:00.221043 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:35:00.221030 2557 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="bd8e0083-3d60-4b95-9ca5-9b54acb813df" path="/var/lib/kubelet/pods/bd8e0083-3d60-4b95-9ca5-9b54acb813df/volumes" Sep 10 13:35:01.060448 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:35:01.060382 2557 patch_prober.go:28] interesting pod/e2e-shared-a-w3-7657ef-openshift-default-85d55f76f6-gw88p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.63:15021/healthz/ready\": dial tcp 10.134.0.63:15021: connect: connection refused" start-of-body= Sep 10 13:35:01.060955 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:35:01.060477 2557 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-7657ef-openshift-default-85d55f76f6-gw88p" podUID="979718af-bb04-448e-8c47-4c82a9adce63" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.63:15021/healthz/ready\": dial tcp 10.134.0.63:15021: connect: connection refused" Sep 10 13:35:02.060347 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:35:02.060307 2557 patch_prober.go:28] interesting pod/e2e-shared-a-w3-7657ef-openshift-default-85d55f76f6-gw88p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.63:15021/healthz/ready\": dial tcp 10.134.0.63:15021: connect: connection refused" start-of-body= Sep 10 13:35:02.060668 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:35:02.060633 2557 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-7657ef-openshift-default-85d55f76f6-gw88p" podUID="979718af-bb04-448e-8c47-4c82a9adce63" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.63:15021/healthz/ready\": dial tcp 10.134.0.63:15021: connect: connection refused" Sep 10 13:35:03.060571 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:35:03.060529 2557 patch_prober.go:28] interesting pod/e2e-shared-a-w3-7657ef-openshift-default-85d55f76f6-gw88p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.63:15021/healthz/ready\": dial tcp 10.134.0.63:15021: connect: connection refused" start-of-body= Sep 10 13:35:03.060905 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:35:03.060867 2557 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-7657ef-openshift-default-85d55f76f6-gw88p" podUID="979718af-bb04-448e-8c47-4c82a9adce63" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.63:15021/healthz/ready\": dial tcp 10.134.0.63:15021: connect: connection refused" Sep 10 13:35:04.060852 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:35:04.060812 2557 patch_prober.go:28] interesting pod/e2e-shared-a-w3-7657ef-openshift-default-85d55f76f6-gw88p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.63:15021/healthz/ready\": dial tcp 10.134.0.63:15021: connect: connection refused" start-of-body= Sep 10 13:35:04.061147 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:35:04.061118 2557 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-7657ef-openshift-default-85d55f76f6-gw88p" podUID="979718af-bb04-448e-8c47-4c82a9adce63" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.63:15021/healthz/ready\": dial tcp 10.134.0.63:15021: connect: connection refused" Sep 10 13:35:05.063765 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:35:05.063736 2557 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w3-7657ef-openshift-default-85d55f76f6-gw88p" Sep 10 13:35:05.064220 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:35:05.064138 2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w3-7657ef-openshift-default-85d55f76f6-gw88p" Sep 10 13:35:05.064736 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:35:05.064720 2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w3-7657ef-openshift-default-85d55f76f6-gw88p" Sep 10 13:35:08.052567 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:35:08.052535 2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-7f8babf4-openshift-default-74f9755b6d-dzdc9"] Sep 10 13:35:08.053004 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:35:08.052757 2557 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-7f8babf4-openshift-default-74f9755b6d-dzdc9" podUID="287603f3-efd3-46a9-bbfe-312f96974f76" containerName="istio-proxy" containerID="cri-o://f279ee7e7781d02a864be4367ebee3bc23385711bfcaa5cacbebd6810c4d56d0" gracePeriod=30 Sep 10 13:35:08.280147 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:35:08.280108 2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w3-7739cb-openshift-default-6764b957b8-zn8rv"] Sep 10 13:35:08.280974 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:35:08.280939 2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bd8e0083-3d60-4b95-9ca5-9b54acb813df" containerName="revoke-keys" Sep 10 13:35:08.280974 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:35:08.280969 2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="bd8e0083-3d60-4b95-9ca5-9b54acb813df" containerName="revoke-keys" Sep 10 13:35:08.281121 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:35:08.280993 2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b23f2560-5aa1-43b0-b0d9-674c6b0eb707" containerName="istio-proxy" Sep 10 13:35:08.281121 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:35:08.281003 2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="b23f2560-5aa1-43b0-b0d9-674c6b0eb707" containerName="istio-proxy" Sep 10 13:35:08.281121 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:35:08.281017 2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3da88b7f-bbfb-4521-acc5-dfe20cacbced" containerName="payload-processing" Sep 10 13:35:08.281121 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:35:08.281026 2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="3da88b7f-bbfb-4521-acc5-dfe20cacbced" containerName="payload-processing" Sep 10 13:35:08.281269 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:35:08.281124 2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="bd8e0083-3d60-4b95-9ca5-9b54acb813df" containerName="revoke-keys" Sep 10 13:35:08.281269 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:35:08.281138 2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="3da88b7f-bbfb-4521-acc5-dfe20cacbced" containerName="payload-processing" Sep 10 13:35:08.285608 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:35:08.285590 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w3-7739cb-openshift-default-6764b957b8-zn8rv" Sep 10 13:35:08.288584 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:35:08.288563 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w3-7739cb-openshift-default-dockercfg-c99mf\"" Sep 10 13:35:08.314867 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:35:08.314802 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-7739cb-openshift-default-6764b957b8-zn8rv"] Sep 10 13:35:08.355014 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:35:08.354987 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3b257e70-49de-40e6-940f-2306f4a81156-istio-token\") pod \"e2e-shared-b-w3-7739cb-openshift-default-6764b957b8-zn8rv\" (UID: \"3b257e70-49de-40e6-940f-2306f4a81156\") " pod="openshift-ingress/e2e-shared-b-w3-7739cb-openshift-default-6764b957b8-zn8rv" Sep 10 13:35:08.355155 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:35:08.355025 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3b257e70-49de-40e6-940f-2306f4a81156-istio-envoy\") pod \"e2e-shared-b-w3-7739cb-openshift-default-6764b957b8-zn8rv\" (UID: \"3b257e70-49de-40e6-940f-2306f4a81156\") " pod="openshift-ingress/e2e-shared-b-w3-7739cb-openshift-default-6764b957b8-zn8rv" Sep 10 13:35:08.355155 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:35:08.355046 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3b257e70-49de-40e6-940f-2306f4a81156-istio-podinfo\") pod \"e2e-shared-b-w3-7739cb-openshift-default-6764b957b8-zn8rv\" (UID: \"3b257e70-49de-40e6-940f-2306f4a81156\") " pod="openshift-ingress/e2e-shared-b-w3-7739cb-openshift-default-6764b957b8-zn8rv" Sep 10 13:35:08.355155 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:35:08.355116 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3b257e70-49de-40e6-940f-2306f4a81156-credential-socket\") pod \"e2e-shared-b-w3-7739cb-openshift-default-6764b957b8-zn8rv\" (UID: \"3b257e70-49de-40e6-940f-2306f4a81156\") " pod="openshift-ingress/e2e-shared-b-w3-7739cb-openshift-default-6764b957b8-zn8rv" Sep 10 13:35:08.355364 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:35:08.355173 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3b257e70-49de-40e6-940f-2306f4a81156-workload-certs\") pod \"e2e-shared-b-w3-7739cb-openshift-default-6764b957b8-zn8rv\" (UID: \"3b257e70-49de-40e6-940f-2306f4a81156\") " pod="openshift-ingress/e2e-shared-b-w3-7739cb-openshift-default-6764b957b8-zn8rv" Sep 10 13:35:08.355364 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:35:08.355286 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tnfhn\" (UniqueName: \"kubernetes.io/projected/3b257e70-49de-40e6-940f-2306f4a81156-kube-api-access-tnfhn\") pod \"e2e-shared-b-w3-7739cb-openshift-default-6764b957b8-zn8rv\" (UID: \"3b257e70-49de-40e6-940f-2306f4a81156\") " pod="openshift-ingress/e2e-shared-b-w3-7739cb-openshift-default-6764b957b8-zn8rv" Sep 10 13:35:08.355364 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:35:08.355329 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3b257e70-49de-40e6-940f-2306f4a81156-workload-socket\") pod \"e2e-shared-b-w3-7739cb-openshift-default-6764b957b8-zn8rv\" (UID: \"3b257e70-49de-40e6-940f-2306f4a81156\") " pod="openshift-ingress/e2e-shared-b-w3-7739cb-openshift-default-6764b957b8-zn8rv" Sep 10 13:35:08.355534 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:35:08.355381 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3b257e70-49de-40e6-940f-2306f4a81156-istio-data\") pod \"e2e-shared-b-w3-7739cb-openshift-default-6764b957b8-zn8rv\" (UID: \"3b257e70-49de-40e6-940f-2306f4a81156\") " pod="openshift-ingress/e2e-shared-b-w3-7739cb-openshift-default-6764b957b8-zn8rv" Sep 10 13:35:08.355586 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:35:08.355525 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3b257e70-49de-40e6-940f-2306f4a81156-istiod-ca-cert\") pod \"e2e-shared-b-w3-7739cb-openshift-default-6764b957b8-zn8rv\" (UID: \"3b257e70-49de-40e6-940f-2306f4a81156\") " pod="openshift-ingress/e2e-shared-b-w3-7739cb-openshift-default-6764b957b8-zn8rv" Sep 10 13:35:08.456290 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:35:08.456255 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3b257e70-49de-40e6-940f-2306f4a81156-workload-socket\") pod \"e2e-shared-b-w3-7739cb-openshift-default-6764b957b8-zn8rv\" (UID: \"3b257e70-49de-40e6-940f-2306f4a81156\") " pod="openshift-ingress/e2e-shared-b-w3-7739cb-openshift-default-6764b957b8-zn8rv" Sep 10 13:35:08.456290 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:35:08.456299 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3b257e70-49de-40e6-940f-2306f4a81156-istio-data\") pod \"e2e-shared-b-w3-7739cb-openshift-default-6764b957b8-zn8rv\" (UID: \"3b257e70-49de-40e6-940f-2306f4a81156\") " pod="openshift-ingress/e2e-shared-b-w3-7739cb-openshift-default-6764b957b8-zn8rv" Sep 10 13:35:08.456590 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:35:08.456339 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3b257e70-49de-40e6-940f-2306f4a81156-istiod-ca-cert\") pod \"e2e-shared-b-w3-7739cb-openshift-default-6764b957b8-zn8rv\" (UID: \"3b257e70-49de-40e6-940f-2306f4a81156\") " pod="openshift-ingress/e2e-shared-b-w3-7739cb-openshift-default-6764b957b8-zn8rv" Sep 10 13:35:08.456590 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:35:08.456371 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3b257e70-49de-40e6-940f-2306f4a81156-istio-token\") pod \"e2e-shared-b-w3-7739cb-openshift-default-6764b957b8-zn8rv\" (UID: \"3b257e70-49de-40e6-940f-2306f4a81156\") " pod="openshift-ingress/e2e-shared-b-w3-7739cb-openshift-default-6764b957b8-zn8rv" Sep 10 13:35:08.456590 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:35:08.456471 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3b257e70-49de-40e6-940f-2306f4a81156-istio-envoy\") pod \"e2e-shared-b-w3-7739cb-openshift-default-6764b957b8-zn8rv\" (UID: \"3b257e70-49de-40e6-940f-2306f4a81156\") " pod="openshift-ingress/e2e-shared-b-w3-7739cb-openshift-default-6764b957b8-zn8rv" Sep 10 13:35:08.456590 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:35:08.456506 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3b257e70-49de-40e6-940f-2306f4a81156-istio-podinfo\") pod \"e2e-shared-b-w3-7739cb-openshift-default-6764b957b8-zn8rv\" (UID: \"3b257e70-49de-40e6-940f-2306f4a81156\") " pod="openshift-ingress/e2e-shared-b-w3-7739cb-openshift-default-6764b957b8-zn8rv" Sep 10 13:35:08.456590 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:35:08.456557 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3b257e70-49de-40e6-940f-2306f4a81156-credential-socket\") pod \"e2e-shared-b-w3-7739cb-openshift-default-6764b957b8-zn8rv\" (UID: \"3b257e70-49de-40e6-940f-2306f4a81156\") " pod="openshift-ingress/e2e-shared-b-w3-7739cb-openshift-default-6764b957b8-zn8rv" Sep 10 13:35:08.456840 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:35:08.456614 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3b257e70-49de-40e6-940f-2306f4a81156-workload-certs\") pod \"e2e-shared-b-w3-7739cb-openshift-default-6764b957b8-zn8rv\" (UID: \"3b257e70-49de-40e6-940f-2306f4a81156\") " pod="openshift-ingress/e2e-shared-b-w3-7739cb-openshift-default-6764b957b8-zn8rv" Sep 10 13:35:08.456840 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:35:08.456677 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tnfhn\" (UniqueName: \"kubernetes.io/projected/3b257e70-49de-40e6-940f-2306f4a81156-kube-api-access-tnfhn\") pod \"e2e-shared-b-w3-7739cb-openshift-default-6764b957b8-zn8rv\" (UID: \"3b257e70-49de-40e6-940f-2306f4a81156\") " pod="openshift-ingress/e2e-shared-b-w3-7739cb-openshift-default-6764b957b8-zn8rv" Sep 10 13:35:08.456840 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:35:08.456745 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3b257e70-49de-40e6-940f-2306f4a81156-workload-socket\") pod \"e2e-shared-b-w3-7739cb-openshift-default-6764b957b8-zn8rv\" (UID: \"3b257e70-49de-40e6-940f-2306f4a81156\") " pod="openshift-ingress/e2e-shared-b-w3-7739cb-openshift-default-6764b957b8-zn8rv" Sep 10 13:35:08.457107 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:35:08.457081 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3b257e70-49de-40e6-940f-2306f4a81156-istiod-ca-cert\") pod \"e2e-shared-b-w3-7739cb-openshift-default-6764b957b8-zn8rv\" (UID: \"3b257e70-49de-40e6-940f-2306f4a81156\") " pod="openshift-ingress/e2e-shared-b-w3-7739cb-openshift-default-6764b957b8-zn8rv" Sep 10 13:35:08.457220 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:35:08.457145 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3b257e70-49de-40e6-940f-2306f4a81156-istio-data\") pod \"e2e-shared-b-w3-7739cb-openshift-default-6764b957b8-zn8rv\" (UID: \"3b257e70-49de-40e6-940f-2306f4a81156\") " pod="openshift-ingress/e2e-shared-b-w3-7739cb-openshift-default-6764b957b8-zn8rv" Sep 10 13:35:08.457327 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:35:08.457308 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3b257e70-49de-40e6-940f-2306f4a81156-workload-certs\") pod \"e2e-shared-b-w3-7739cb-openshift-default-6764b957b8-zn8rv\" (UID: \"3b257e70-49de-40e6-940f-2306f4a81156\") " pod="openshift-ingress/e2e-shared-b-w3-7739cb-openshift-default-6764b957b8-zn8rv" Sep 10 13:35:08.457327 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:35:08.457316 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3b257e70-49de-40e6-940f-2306f4a81156-credential-socket\") pod \"e2e-shared-b-w3-7739cb-openshift-default-6764b957b8-zn8rv\" (UID: \"3b257e70-49de-40e6-940f-2306f4a81156\") " pod="openshift-ingress/e2e-shared-b-w3-7739cb-openshift-default-6764b957b8-zn8rv" Sep 10 13:35:08.458618 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:35:08.458595 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3b257e70-49de-40e6-940f-2306f4a81156-istio-envoy\") pod \"e2e-shared-b-w3-7739cb-openshift-default-6764b957b8-zn8rv\" (UID: \"3b257e70-49de-40e6-940f-2306f4a81156\") " pod="openshift-ingress/e2e-shared-b-w3-7739cb-openshift-default-6764b957b8-zn8rv" Sep 10 13:35:08.458982 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:35:08.458964 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3b257e70-49de-40e6-940f-2306f4a81156-istio-podinfo\") pod \"e2e-shared-b-w3-7739cb-openshift-default-6764b957b8-zn8rv\" (UID: \"3b257e70-49de-40e6-940f-2306f4a81156\") " pod="openshift-ingress/e2e-shared-b-w3-7739cb-openshift-default-6764b957b8-zn8rv" Sep 10 13:35:08.490361 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:35:08.490338 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tnfhn\" (UniqueName: \"kubernetes.io/projected/3b257e70-49de-40e6-940f-2306f4a81156-kube-api-access-tnfhn\") pod \"e2e-shared-b-w3-7739cb-openshift-default-6764b957b8-zn8rv\" (UID: \"3b257e70-49de-40e6-940f-2306f4a81156\") " pod="openshift-ingress/e2e-shared-b-w3-7739cb-openshift-default-6764b957b8-zn8rv" Sep 10 13:35:08.496956 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:35:08.496934 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3b257e70-49de-40e6-940f-2306f4a81156-istio-token\") pod \"e2e-shared-b-w3-7739cb-openshift-default-6764b957b8-zn8rv\" (UID: \"3b257e70-49de-40e6-940f-2306f4a81156\") " pod="openshift-ingress/e2e-shared-b-w3-7739cb-openshift-default-6764b957b8-zn8rv" Sep 10 13:35:08.596483 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:35:08.596369 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w3-7739cb-openshift-default-6764b957b8-zn8rv" Sep 10 13:35:08.889753 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:35:08.889726 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-7739cb-openshift-default-6764b957b8-zn8rv"] Sep 10 13:35:08.891936 ip-10-0-139-174 kubenswrapper[2557]: W0910 13:35:08.891900 2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3b257e70_49de_40e6_940f_2306f4a81156.slice/crio-d84805aaa84bd9849eef06220f5bfc6966b1c4bd67adbe43e0f07a4ee7a069d4 WatchSource:0}: Error finding container d84805aaa84bd9849eef06220f5bfc6966b1c4bd67adbe43e0f07a4ee7a069d4: Status 404 returned error can't find the container with id d84805aaa84bd9849eef06220f5bfc6966b1c4bd67adbe43e0f07a4ee7a069d4 Sep 10 13:35:08.894168 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:35:08.894136 2557 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892152Ki","pods":"250"} Sep 10 13:35:08.894255 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:35:08.894207 2557 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892152Ki","pods":"250"} Sep 10 13:35:08.894255 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:35:08.894235 2557 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892152Ki","pods":"250"} Sep 10 13:35:09.362182 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:35:09.362142 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-7739cb-openshift-default-6764b957b8-zn8rv" event={"ID":"3b257e70-49de-40e6-940f-2306f4a81156","Type":"ContainerStarted","Data":"140c2c6a411ab954abe37e465cf3f295d26a01c3b2ae15ca0a7281a252ffcf9e"} Sep 10 13:35:09.362591 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:35:09.362200 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-7739cb-openshift-default-6764b957b8-zn8rv" event={"ID":"3b257e70-49de-40e6-940f-2306f4a81156","Type":"ContainerStarted","Data":"d84805aaa84bd9849eef06220f5bfc6966b1c4bd67adbe43e0f07a4ee7a069d4"} Sep 10 13:35:09.398572 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:35:09.398520 2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w3-7739cb-openshift-default-6764b957b8-zn8rv" podStartSLOduration=1.398505733 podStartE2EDuration="1.398505733s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 13:35:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 13:35:09.397597266 +0000 UTC m=+1069.704591679" watchObservedRunningTime="2026-09-10 13:35:09.398505733 +0000 UTC m=+1069.705500216" Sep 10 13:35:09.597302 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:35:09.597268 2557 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w3-7739cb-openshift-default-6764b957b8-zn8rv" Sep 10 13:35:09.602062 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:35:09.602042 2557 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w3-7739cb-openshift-default-6764b957b8-zn8rv" Sep 10 13:35:10.368199 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:35:10.368160 2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w3-7739cb-openshift-default-6764b957b8-zn8rv" Sep 10 13:35:10.369104 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:35:10.369081 2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w3-7739cb-openshift-default-6764b957b8-zn8rv" Sep 10 13:35:13.294916 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:35:13.294893 2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-7f8babf4-openshift-default-74f9755b6d-dzdc9" Sep 10 13:35:13.386667 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:35:13.386583 2557 generic.go:356] "Generic (PLEG): container finished" podID="287603f3-efd3-46a9-bbfe-312f96974f76" containerID="f279ee7e7781d02a864be4367ebee3bc23385711bfcaa5cacbebd6810c4d56d0" exitCode=0 Sep 10 13:35:13.386667 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:35:13.386627 2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-7f8babf4-openshift-default-74f9755b6d-dzdc9" Sep 10 13:35:13.386667 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:35:13.386661 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-7f8babf4-openshift-default-74f9755b6d-dzdc9" event={"ID":"287603f3-efd3-46a9-bbfe-312f96974f76","Type":"ContainerDied","Data":"f279ee7e7781d02a864be4367ebee3bc23385711bfcaa5cacbebd6810c4d56d0"} Sep 10 13:35:13.386910 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:35:13.386714 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-7f8babf4-openshift-default-74f9755b6d-dzdc9" event={"ID":"287603f3-efd3-46a9-bbfe-312f96974f76","Type":"ContainerDied","Data":"3cf472f37453b04ddf18099305582232e361d016f702d73dec0830ab36ea23f1"} Sep 10 13:35:13.386910 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:35:13.386742 2557 scope.go:117] "RemoveContainer" containerID="f279ee7e7781d02a864be4367ebee3bc23385711bfcaa5cacbebd6810c4d56d0" Sep 10 13:35:13.395884 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:35:13.395867 2557 scope.go:117] "RemoveContainer" containerID="7271da9c01225af01e6db751f4c8a293d3d8f17c252e0722d26916d6c4469885" Sep 10 13:35:13.399076 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:35:13.399058 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/287603f3-efd3-46a9-bbfe-312f96974f76-istio-data\") pod \"287603f3-efd3-46a9-bbfe-312f96974f76\" (UID: \"287603f3-efd3-46a9-bbfe-312f96974f76\") " Sep 10 13:35:13.399165 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:35:13.399085 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/287603f3-efd3-46a9-bbfe-312f96974f76-workload-certs\") pod \"287603f3-efd3-46a9-bbfe-312f96974f76\" (UID: \"287603f3-efd3-46a9-bbfe-312f96974f76\") " Sep 10 13:35:13.399165 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:35:13.399122 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/287603f3-efd3-46a9-bbfe-312f96974f76-workload-socket\") pod \"287603f3-efd3-46a9-bbfe-312f96974f76\" (UID: \"287603f3-efd3-46a9-bbfe-312f96974f76\") " Sep 10 13:35:13.399268 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:35:13.399166 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/287603f3-efd3-46a9-bbfe-312f96974f76-istio-token\") pod \"287603f3-efd3-46a9-bbfe-312f96974f76\" (UID: \"287603f3-efd3-46a9-bbfe-312f96974f76\") " Sep 10 13:35:13.399268 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:35:13.399209 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/287603f3-efd3-46a9-bbfe-312f96974f76-istio-envoy\") pod \"287603f3-efd3-46a9-bbfe-312f96974f76\" (UID: \"287603f3-efd3-46a9-bbfe-312f96974f76\") " Sep 10 13:35:13.399268 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:35:13.399239 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/287603f3-efd3-46a9-bbfe-312f96974f76-istio-podinfo\") pod \"287603f3-efd3-46a9-bbfe-312f96974f76\" (UID: \"287603f3-efd3-46a9-bbfe-312f96974f76\") " Sep 10 13:35:13.399433 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:35:13.399271 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/287603f3-efd3-46a9-bbfe-312f96974f76-istiod-ca-cert\") pod \"287603f3-efd3-46a9-bbfe-312f96974f76\" (UID: \"287603f3-efd3-46a9-bbfe-312f96974f76\") " Sep 10 13:35:13.399433 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:35:13.399322 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/287603f3-efd3-46a9-bbfe-312f96974f76-istio-data" (OuterVolumeSpecName: "istio-data") pod "287603f3-efd3-46a9-bbfe-312f96974f76" (UID: "287603f3-efd3-46a9-bbfe-312f96974f76"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 13:35:13.399433 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:35:13.399339 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/287603f3-efd3-46a9-bbfe-312f96974f76-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "287603f3-efd3-46a9-bbfe-312f96974f76" (UID: "287603f3-efd3-46a9-bbfe-312f96974f76"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 13:35:13.399433 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:35:13.399356 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/287603f3-efd3-46a9-bbfe-312f96974f76-credential-socket\") pod \"287603f3-efd3-46a9-bbfe-312f96974f76\" (UID: \"287603f3-efd3-46a9-bbfe-312f96974f76\") " Sep 10 13:35:13.399653 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:35:13.399438 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qn5rh\" (UniqueName: \"kubernetes.io/projected/287603f3-efd3-46a9-bbfe-312f96974f76-kube-api-access-qn5rh\") pod \"287603f3-efd3-46a9-bbfe-312f96974f76\" (UID: \"287603f3-efd3-46a9-bbfe-312f96974f76\") " Sep 10 13:35:13.399653 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:35:13.399461 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/287603f3-efd3-46a9-bbfe-312f96974f76-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "287603f3-efd3-46a9-bbfe-312f96974f76" (UID: "287603f3-efd3-46a9-bbfe-312f96974f76"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 13:35:13.399918 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:35:13.399662 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/287603f3-efd3-46a9-bbfe-312f96974f76-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "287603f3-efd3-46a9-bbfe-312f96974f76" (UID: "287603f3-efd3-46a9-bbfe-312f96974f76"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 13:35:13.399918 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:35:13.399723 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/287603f3-efd3-46a9-bbfe-312f96974f76-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "287603f3-efd3-46a9-bbfe-312f96974f76" (UID: "287603f3-efd3-46a9-bbfe-312f96974f76"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 10 13:35:13.400033 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:35:13.399923 2557 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/287603f3-efd3-46a9-bbfe-312f96974f76-istio-data\") on node \"ip-10-0-139-174.ec2.internal\" DevicePath \"\"" Sep 10 13:35:13.400033 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:35:13.399941 2557 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/287603f3-efd3-46a9-bbfe-312f96974f76-workload-certs\") on node \"ip-10-0-139-174.ec2.internal\" DevicePath \"\"" Sep 10 13:35:13.400033 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:35:13.399955 2557 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/287603f3-efd3-46a9-bbfe-312f96974f76-workload-socket\") on node \"ip-10-0-139-174.ec2.internal\" DevicePath \"\"" Sep 10 13:35:13.400033 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:35:13.399968 2557 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/287603f3-efd3-46a9-bbfe-312f96974f76-istiod-ca-cert\") on node \"ip-10-0-139-174.ec2.internal\" DevicePath \"\"" Sep 10 13:35:13.400033 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:35:13.399982 2557 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/287603f3-efd3-46a9-bbfe-312f96974f76-credential-socket\") on node \"ip-10-0-139-174.ec2.internal\" DevicePath \"\"" Sep 10 13:35:13.401729 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:35:13.401706 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/287603f3-efd3-46a9-bbfe-312f96974f76-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "287603f3-efd3-46a9-bbfe-312f96974f76" (UID: "287603f3-efd3-46a9-bbfe-312f96974f76"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Sep 10 13:35:13.401931 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:35:13.401894 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/287603f3-efd3-46a9-bbfe-312f96974f76-istio-token" (OuterVolumeSpecName: "istio-token") pod "287603f3-efd3-46a9-bbfe-312f96974f76" (UID: "287603f3-efd3-46a9-bbfe-312f96974f76"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 10 13:35:13.401931 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:35:13.401897 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/287603f3-efd3-46a9-bbfe-312f96974f76-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "287603f3-efd3-46a9-bbfe-312f96974f76" (UID: "287603f3-efd3-46a9-bbfe-312f96974f76"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 13:35:13.402037 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:35:13.401999 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/287603f3-efd3-46a9-bbfe-312f96974f76-kube-api-access-qn5rh" (OuterVolumeSpecName: "kube-api-access-qn5rh") pod "287603f3-efd3-46a9-bbfe-312f96974f76" (UID: "287603f3-efd3-46a9-bbfe-312f96974f76"). InnerVolumeSpecName "kube-api-access-qn5rh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 10 13:35:13.405534 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:35:13.405514 2557 scope.go:117] "RemoveContainer" containerID="f279ee7e7781d02a864be4367ebee3bc23385711bfcaa5cacbebd6810c4d56d0" Sep 10 13:35:13.405831 ip-10-0-139-174 kubenswrapper[2557]: E0910 13:35:13.405811 2557 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f279ee7e7781d02a864be4367ebee3bc23385711bfcaa5cacbebd6810c4d56d0\": container with ID starting with f279ee7e7781d02a864be4367ebee3bc23385711bfcaa5cacbebd6810c4d56d0 not found: ID does not exist" containerID="f279ee7e7781d02a864be4367ebee3bc23385711bfcaa5cacbebd6810c4d56d0" Sep 10 13:35:13.405886 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:35:13.405838 2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f279ee7e7781d02a864be4367ebee3bc23385711bfcaa5cacbebd6810c4d56d0"} err="failed to get container status \"f279ee7e7781d02a864be4367ebee3bc23385711bfcaa5cacbebd6810c4d56d0\": rpc error: code = NotFound desc = could not find container \"f279ee7e7781d02a864be4367ebee3bc23385711bfcaa5cacbebd6810c4d56d0\": container with ID starting with f279ee7e7781d02a864be4367ebee3bc23385711bfcaa5cacbebd6810c4d56d0 not found: ID does not exist" Sep 10 13:35:13.405886 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:35:13.405855 2557 scope.go:117] "RemoveContainer" containerID="7271da9c01225af01e6db751f4c8a293d3d8f17c252e0722d26916d6c4469885" Sep 10 13:35:13.406086 ip-10-0-139-174 kubenswrapper[2557]: E0910 13:35:13.406071 2557 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7271da9c01225af01e6db751f4c8a293d3d8f17c252e0722d26916d6c4469885\": container with ID starting with 7271da9c01225af01e6db751f4c8a293d3d8f17c252e0722d26916d6c4469885 not found: ID does not exist" containerID="7271da9c01225af01e6db751f4c8a293d3d8f17c252e0722d26916d6c4469885" Sep 10 13:35:13.406131 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:35:13.406091 2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7271da9c01225af01e6db751f4c8a293d3d8f17c252e0722d26916d6c4469885"} err="failed to get container status \"7271da9c01225af01e6db751f4c8a293d3d8f17c252e0722d26916d6c4469885\": rpc error: code = NotFound desc = could not find container \"7271da9c01225af01e6db751f4c8a293d3d8f17c252e0722d26916d6c4469885\": container with ID starting with 7271da9c01225af01e6db751f4c8a293d3d8f17c252e0722d26916d6c4469885 not found: ID does not exist" Sep 10 13:35:13.500734 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:35:13.500710 2557 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/287603f3-efd3-46a9-bbfe-312f96974f76-istio-token\") on node \"ip-10-0-139-174.ec2.internal\" DevicePath \"\"" Sep 10 13:35:13.500734 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:35:13.500731 2557 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/287603f3-efd3-46a9-bbfe-312f96974f76-istio-envoy\") on node \"ip-10-0-139-174.ec2.internal\" DevicePath \"\"" Sep 10 13:35:13.500877 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:35:13.500739 2557 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/287603f3-efd3-46a9-bbfe-312f96974f76-istio-podinfo\") on node \"ip-10-0-139-174.ec2.internal\" DevicePath \"\"" Sep 10 13:35:13.501152 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:35:13.501126 2557 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qn5rh\" (UniqueName: \"kubernetes.io/projected/287603f3-efd3-46a9-bbfe-312f96974f76-kube-api-access-qn5rh\") on node \"ip-10-0-139-174.ec2.internal\" DevicePath \"\"" Sep 10 13:35:13.724019 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:35:13.723996 2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-7f8babf4-openshift-default-74f9755b6d-dzdc9"] Sep 10 13:35:13.730496 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:35:13.730472 2557 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-7f8babf4-openshift-default-74f9755b6d-dzdc9"] Sep 10 13:35:14.220590 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:35:14.220556 2557 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="287603f3-efd3-46a9-bbfe-312f96974f76" path="/var/lib/kubelet/pods/287603f3-efd3-46a9-bbfe-312f96974f76/volumes" Sep 10 13:35:14.283908 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:35:14.283873 2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-7739cb-79cc8c98c-gn8pb"] Sep 10 13:35:14.284296 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:35:14.284281 2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="287603f3-efd3-46a9-bbfe-312f96974f76" containerName="istio-proxy" Sep 10 13:35:14.284338 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:35:14.284300 2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="287603f3-efd3-46a9-bbfe-312f96974f76" containerName="istio-proxy" Sep 10 13:35:14.284338 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:35:14.284314 2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="287603f3-efd3-46a9-bbfe-312f96974f76" containerName="istio-proxy" Sep 10 13:35:14.284338 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:35:14.284321 2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="287603f3-efd3-46a9-bbfe-312f96974f76" containerName="istio-proxy" Sep 10 13:35:14.284459 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:35:14.284396 2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="287603f3-efd3-46a9-bbfe-312f96974f76" containerName="istio-proxy" Sep 10 13:35:14.284459 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:35:14.284406 2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="287603f3-efd3-46a9-bbfe-312f96974f76" containerName="istio-proxy" Sep 10 13:35:14.289059 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:35:14.289040 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-7739cb-79cc8c98c-gn8pb" Sep 10 13:35:14.292270 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:35:14.292251 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-b-w3-7739cb-dockercfg-8sfwq\"" Sep 10 13:35:14.292395 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:35:14.292301 2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-w3-7739cb\"" Sep 10 13:35:14.306468 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:35:14.306443 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-7739cb-79cc8c98c-gn8pb"] Sep 10 13:35:14.319967 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:35:14.319937 2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-7739cb-5b465cd6d6-snwfr"] Sep 10 13:35:14.323476 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:35:14.323458 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-7739cb-5b465cd6d6-snwfr" Sep 10 13:35:14.342164 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:35:14.342142 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-7739cb-5b465cd6d6-snwfr"] Sep 10 13:35:14.411149 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:35:14.411117 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e14e9221-9cf2-44a5-8b2c-b995c364cead-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w3-7739cb-79cc8c98c-gn8pb\" (UID: \"e14e9221-9cf2-44a5-8b2c-b995c364cead\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-7739cb-79cc8c98c-gn8pb" Sep 10 13:35:14.411149 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:35:14.411151 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c6fa8034-38ba-4d3a-a60a-3355bc14ed94-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w3-7739cb-5b465cd6d6-snwfr\" (UID: \"c6fa8034-38ba-4d3a-a60a-3355bc14ed94\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-7739cb-5b465cd6d6-snwfr" Sep 10 13:35:14.411359 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:35:14.411173 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hrtxp\" (UniqueName: \"kubernetes.io/projected/e14e9221-9cf2-44a5-8b2c-b995c364cead-kube-api-access-hrtxp\") pod \"payload-processing-e2e-shared-b-w3-7739cb-79cc8c98c-gn8pb\" (UID: \"e14e9221-9cf2-44a5-8b2c-b995c364cead\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-7739cb-79cc8c98c-gn8pb" Sep 10 13:35:14.411359 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:35:14.411276 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-82tzg\" (UniqueName: \"kubernetes.io/projected/c6fa8034-38ba-4d3a-a60a-3355bc14ed94-kube-api-access-82tzg\") pod \"payload-pre-processing-e2e-shared-b-w3-7739cb-5b465cd6d6-snwfr\" (UID: \"c6fa8034-38ba-4d3a-a60a-3355bc14ed94\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-7739cb-5b465cd6d6-snwfr" Sep 10 13:35:14.512991 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:35:14.512907 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-82tzg\" (UniqueName: \"kubernetes.io/projected/c6fa8034-38ba-4d3a-a60a-3355bc14ed94-kube-api-access-82tzg\") pod \"payload-pre-processing-e2e-shared-b-w3-7739cb-5b465cd6d6-snwfr\" (UID: \"c6fa8034-38ba-4d3a-a60a-3355bc14ed94\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-7739cb-5b465cd6d6-snwfr" Sep 10 13:35:14.513127 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:35:14.513078 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e14e9221-9cf2-44a5-8b2c-b995c364cead-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w3-7739cb-79cc8c98c-gn8pb\" (UID: \"e14e9221-9cf2-44a5-8b2c-b995c364cead\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-7739cb-79cc8c98c-gn8pb" Sep 10 13:35:14.513127 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:35:14.513115 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c6fa8034-38ba-4d3a-a60a-3355bc14ed94-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w3-7739cb-5b465cd6d6-snwfr\" (UID: \"c6fa8034-38ba-4d3a-a60a-3355bc14ed94\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-7739cb-5b465cd6d6-snwfr" Sep 10 13:35:14.513211 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:35:14.513150 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hrtxp\" (UniqueName: \"kubernetes.io/projected/e14e9221-9cf2-44a5-8b2c-b995c364cead-kube-api-access-hrtxp\") pod \"payload-processing-e2e-shared-b-w3-7739cb-79cc8c98c-gn8pb\" (UID: \"e14e9221-9cf2-44a5-8b2c-b995c364cead\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-7739cb-79cc8c98c-gn8pb" Sep 10 13:35:14.514027 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:35:14.514005 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c6fa8034-38ba-4d3a-a60a-3355bc14ed94-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w3-7739cb-5b465cd6d6-snwfr\" (UID: \"c6fa8034-38ba-4d3a-a60a-3355bc14ed94\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-7739cb-5b465cd6d6-snwfr" Sep 10 13:35:14.514069 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:35:14.514005 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e14e9221-9cf2-44a5-8b2c-b995c364cead-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w3-7739cb-79cc8c98c-gn8pb\" (UID: \"e14e9221-9cf2-44a5-8b2c-b995c364cead\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-7739cb-79cc8c98c-gn8pb" Sep 10 13:35:14.526358 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:35:14.526329 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-82tzg\" (UniqueName: \"kubernetes.io/projected/c6fa8034-38ba-4d3a-a60a-3355bc14ed94-kube-api-access-82tzg\") pod \"payload-pre-processing-e2e-shared-b-w3-7739cb-5b465cd6d6-snwfr\" (UID: \"c6fa8034-38ba-4d3a-a60a-3355bc14ed94\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-7739cb-5b465cd6d6-snwfr" Sep 10 13:35:14.526965 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:35:14.526944 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hrtxp\" (UniqueName: \"kubernetes.io/projected/e14e9221-9cf2-44a5-8b2c-b995c364cead-kube-api-access-hrtxp\") pod \"payload-processing-e2e-shared-b-w3-7739cb-79cc8c98c-gn8pb\" (UID: \"e14e9221-9cf2-44a5-8b2c-b995c364cead\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-7739cb-79cc8c98c-gn8pb" Sep 10 13:35:14.599274 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:35:14.599247 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-7739cb-79cc8c98c-gn8pb" Sep 10 13:35:14.634370 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:35:14.634340 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-7739cb-5b465cd6d6-snwfr" Sep 10 13:35:14.889469 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:35:14.889445 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-7739cb-79cc8c98c-gn8pb"] Sep 10 13:35:14.891106 ip-10-0-139-174 kubenswrapper[2557]: W0910 13:35:14.891074 2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode14e9221_9cf2_44a5_8b2c_b995c364cead.slice/crio-aaa02d262fb9d4f2c562520254b40fa9cc6459d208ec8fcbd8475055465910c1 WatchSource:0}: Error finding container aaa02d262fb9d4f2c562520254b40fa9cc6459d208ec8fcbd8475055465910c1: Status 404 returned error can't find the container with id aaa02d262fb9d4f2c562520254b40fa9cc6459d208ec8fcbd8475055465910c1 Sep 10 13:35:14.927990 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:35:14.927966 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-7739cb-5b465cd6d6-snwfr"] Sep 10 13:35:14.931638 ip-10-0-139-174 kubenswrapper[2557]: W0910 13:35:14.931610 2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc6fa8034_38ba_4d3a_a60a_3355bc14ed94.slice/crio-fbf3da115e0d1fc08cc9f48f8ad13730c0bac0ba95bb86a2efdc5dbf61179f20 WatchSource:0}: Error finding container fbf3da115e0d1fc08cc9f48f8ad13730c0bac0ba95bb86a2efdc5dbf61179f20: Status 404 returned error can't find the container with id fbf3da115e0d1fc08cc9f48f8ad13730c0bac0ba95bb86a2efdc5dbf61179f20 Sep 10 13:35:15.402241 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:35:15.402201 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-7739cb-5b465cd6d6-snwfr" event={"ID":"c6fa8034-38ba-4d3a-a60a-3355bc14ed94","Type":"ContainerStarted","Data":"31bf4dae536c57381a93f4eb814e6667d414830a9b3f40415ae2ae9b677fb044"} Sep 10 13:35:15.402716 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:35:15.402261 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-7739cb-5b465cd6d6-snwfr" event={"ID":"c6fa8034-38ba-4d3a-a60a-3355bc14ed94","Type":"ContainerStarted","Data":"fbf3da115e0d1fc08cc9f48f8ad13730c0bac0ba95bb86a2efdc5dbf61179f20"} Sep 10 13:35:15.402716 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:35:15.402386 2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-7739cb-5b465cd6d6-snwfr" Sep 10 13:35:15.404062 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:35:15.404035 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-7739cb-79cc8c98c-gn8pb" event={"ID":"e14e9221-9cf2-44a5-8b2c-b995c364cead","Type":"ContainerStarted","Data":"0fde2177917ece876f2de6d674eadbdfc207b421afcab6f9dd08fb468133b223"} Sep 10 13:35:15.404170 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:35:15.404088 2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-7739cb-79cc8c98c-gn8pb" Sep 10 13:35:15.404170 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:35:15.404111 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-7739cb-79cc8c98c-gn8pb" event={"ID":"e14e9221-9cf2-44a5-8b2c-b995c364cead","Type":"ContainerStarted","Data":"aaa02d262fb9d4f2c562520254b40fa9cc6459d208ec8fcbd8475055465910c1"} Sep 10 13:35:15.435494 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:35:15.435445 2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-7739cb-5b465cd6d6-snwfr" podStartSLOduration=1.435413224 podStartE2EDuration="1.435413224s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 13:35:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 13:35:15.435090348 +0000 UTC m=+1075.742084762" watchObservedRunningTime="2026-09-10 13:35:15.435413224 +0000 UTC m=+1075.742407661" Sep 10 13:35:15.480561 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:35:15.480507 2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-7739cb-79cc8c98c-gn8pb" podStartSLOduration=1.4804936419999999 podStartE2EDuration="1.480493642s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 13:35:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 13:35:15.477592349 +0000 UTC m=+1075.784586763" watchObservedRunningTime="2026-09-10 13:35:15.480493642 +0000 UTC m=+1075.787488055" Sep 10 13:35:25.691895 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:35:25.691862 2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w2-c165a4-openshift-default-5b5c6d994b-g4wbc"] Sep 10 13:35:25.697234 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:35:25.697203 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-c165a4-openshift-default-5b5c6d994b-g4wbc" Sep 10 13:35:25.699682 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:35:25.699658 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w2-c165a4-openshift-default-dockercfg-8jtkb\"" Sep 10 13:35:25.764984 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:35:25.764951 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-c165a4-openshift-default-5b5c6d994b-g4wbc"] Sep 10 13:35:25.829137 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:35:25.829096 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b65e81d0-d6df-44d2-a87a-04b4a58eb154-istiod-ca-cert\") pod \"e2e-shared-a-w2-c165a4-openshift-default-5b5c6d994b-g4wbc\" (UID: \"b65e81d0-d6df-44d2-a87a-04b4a58eb154\") " pod="openshift-ingress/e2e-shared-a-w2-c165a4-openshift-default-5b5c6d994b-g4wbc" Sep 10 13:35:25.829363 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:35:25.829163 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b65e81d0-d6df-44d2-a87a-04b4a58eb154-workload-socket\") pod \"e2e-shared-a-w2-c165a4-openshift-default-5b5c6d994b-g4wbc\" (UID: \"b65e81d0-d6df-44d2-a87a-04b4a58eb154\") " pod="openshift-ingress/e2e-shared-a-w2-c165a4-openshift-default-5b5c6d994b-g4wbc" Sep 10 13:35:25.829363 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:35:25.829228 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b65e81d0-d6df-44d2-a87a-04b4a58eb154-istio-podinfo\") pod \"e2e-shared-a-w2-c165a4-openshift-default-5b5c6d994b-g4wbc\" (UID: \"b65e81d0-d6df-44d2-a87a-04b4a58eb154\") " pod="openshift-ingress/e2e-shared-a-w2-c165a4-openshift-default-5b5c6d994b-g4wbc" Sep 10 13:35:25.829363 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:35:25.829286 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b65e81d0-d6df-44d2-a87a-04b4a58eb154-workload-certs\") pod \"e2e-shared-a-w2-c165a4-openshift-default-5b5c6d994b-g4wbc\" (UID: \"b65e81d0-d6df-44d2-a87a-04b4a58eb154\") " pod="openshift-ingress/e2e-shared-a-w2-c165a4-openshift-default-5b5c6d994b-g4wbc" Sep 10 13:35:25.829363 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:35:25.829315 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2gvt2\" (UniqueName: \"kubernetes.io/projected/b65e81d0-d6df-44d2-a87a-04b4a58eb154-kube-api-access-2gvt2\") pod \"e2e-shared-a-w2-c165a4-openshift-default-5b5c6d994b-g4wbc\" (UID: \"b65e81d0-d6df-44d2-a87a-04b4a58eb154\") " pod="openshift-ingress/e2e-shared-a-w2-c165a4-openshift-default-5b5c6d994b-g4wbc" Sep 10 13:35:25.829554 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:35:25.829372 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b65e81d0-d6df-44d2-a87a-04b4a58eb154-istio-data\") pod \"e2e-shared-a-w2-c165a4-openshift-default-5b5c6d994b-g4wbc\" (UID: \"b65e81d0-d6df-44d2-a87a-04b4a58eb154\") " pod="openshift-ingress/e2e-shared-a-w2-c165a4-openshift-default-5b5c6d994b-g4wbc" Sep 10 13:35:25.829554 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:35:25.829400 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b65e81d0-d6df-44d2-a87a-04b4a58eb154-istio-envoy\") pod \"e2e-shared-a-w2-c165a4-openshift-default-5b5c6d994b-g4wbc\" (UID: \"b65e81d0-d6df-44d2-a87a-04b4a58eb154\") " pod="openshift-ingress/e2e-shared-a-w2-c165a4-openshift-default-5b5c6d994b-g4wbc" Sep 10 13:35:25.829554 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:35:25.829486 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b65e81d0-d6df-44d2-a87a-04b4a58eb154-credential-socket\") pod \"e2e-shared-a-w2-c165a4-openshift-default-5b5c6d994b-g4wbc\" (UID: \"b65e81d0-d6df-44d2-a87a-04b4a58eb154\") " pod="openshift-ingress/e2e-shared-a-w2-c165a4-openshift-default-5b5c6d994b-g4wbc" Sep 10 13:35:25.829657 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:35:25.829611 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b65e81d0-d6df-44d2-a87a-04b4a58eb154-istio-token\") pod \"e2e-shared-a-w2-c165a4-openshift-default-5b5c6d994b-g4wbc\" (UID: \"b65e81d0-d6df-44d2-a87a-04b4a58eb154\") " pod="openshift-ingress/e2e-shared-a-w2-c165a4-openshift-default-5b5c6d994b-g4wbc" Sep 10 13:35:25.930900 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:35:25.930863 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b65e81d0-d6df-44d2-a87a-04b4a58eb154-istiod-ca-cert\") pod \"e2e-shared-a-w2-c165a4-openshift-default-5b5c6d994b-g4wbc\" (UID: \"b65e81d0-d6df-44d2-a87a-04b4a58eb154\") " pod="openshift-ingress/e2e-shared-a-w2-c165a4-openshift-default-5b5c6d994b-g4wbc" Sep 10 13:35:25.931113 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:35:25.930948 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b65e81d0-d6df-44d2-a87a-04b4a58eb154-workload-socket\") pod \"e2e-shared-a-w2-c165a4-openshift-default-5b5c6d994b-g4wbc\" (UID: \"b65e81d0-d6df-44d2-a87a-04b4a58eb154\") " pod="openshift-ingress/e2e-shared-a-w2-c165a4-openshift-default-5b5c6d994b-g4wbc" Sep 10 13:35:25.931113 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:35:25.930992 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b65e81d0-d6df-44d2-a87a-04b4a58eb154-istio-podinfo\") pod \"e2e-shared-a-w2-c165a4-openshift-default-5b5c6d994b-g4wbc\" (UID: \"b65e81d0-d6df-44d2-a87a-04b4a58eb154\") " pod="openshift-ingress/e2e-shared-a-w2-c165a4-openshift-default-5b5c6d994b-g4wbc" Sep 10 13:35:25.931113 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:35:25.931027 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b65e81d0-d6df-44d2-a87a-04b4a58eb154-workload-certs\") pod \"e2e-shared-a-w2-c165a4-openshift-default-5b5c6d994b-g4wbc\" (UID: \"b65e81d0-d6df-44d2-a87a-04b4a58eb154\") " pod="openshift-ingress/e2e-shared-a-w2-c165a4-openshift-default-5b5c6d994b-g4wbc" Sep 10 13:35:25.931113 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:35:25.931054 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2gvt2\" (UniqueName: \"kubernetes.io/projected/b65e81d0-d6df-44d2-a87a-04b4a58eb154-kube-api-access-2gvt2\") pod \"e2e-shared-a-w2-c165a4-openshift-default-5b5c6d994b-g4wbc\" (UID: \"b65e81d0-d6df-44d2-a87a-04b4a58eb154\") " pod="openshift-ingress/e2e-shared-a-w2-c165a4-openshift-default-5b5c6d994b-g4wbc" Sep 10 13:35:25.931113 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:35:25.931078 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b65e81d0-d6df-44d2-a87a-04b4a58eb154-istio-data\") pod \"e2e-shared-a-w2-c165a4-openshift-default-5b5c6d994b-g4wbc\" (UID: \"b65e81d0-d6df-44d2-a87a-04b4a58eb154\") " pod="openshift-ingress/e2e-shared-a-w2-c165a4-openshift-default-5b5c6d994b-g4wbc" Sep 10 13:35:25.931113 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:35:25.931103 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b65e81d0-d6df-44d2-a87a-04b4a58eb154-istio-envoy\") pod \"e2e-shared-a-w2-c165a4-openshift-default-5b5c6d994b-g4wbc\" (UID: \"b65e81d0-d6df-44d2-a87a-04b4a58eb154\") " pod="openshift-ingress/e2e-shared-a-w2-c165a4-openshift-default-5b5c6d994b-g4wbc" Sep 10 13:35:25.931412 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:35:25.931135 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b65e81d0-d6df-44d2-a87a-04b4a58eb154-credential-socket\") pod \"e2e-shared-a-w2-c165a4-openshift-default-5b5c6d994b-g4wbc\" (UID: \"b65e81d0-d6df-44d2-a87a-04b4a58eb154\") " pod="openshift-ingress/e2e-shared-a-w2-c165a4-openshift-default-5b5c6d994b-g4wbc" Sep 10 13:35:25.931412 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:35:25.931243 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b65e81d0-d6df-44d2-a87a-04b4a58eb154-istio-token\") pod \"e2e-shared-a-w2-c165a4-openshift-default-5b5c6d994b-g4wbc\" (UID: \"b65e81d0-d6df-44d2-a87a-04b4a58eb154\") " pod="openshift-ingress/e2e-shared-a-w2-c165a4-openshift-default-5b5c6d994b-g4wbc" Sep 10 13:35:25.931412 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:35:25.931365 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b65e81d0-d6df-44d2-a87a-04b4a58eb154-workload-socket\") pod \"e2e-shared-a-w2-c165a4-openshift-default-5b5c6d994b-g4wbc\" (UID: \"b65e81d0-d6df-44d2-a87a-04b4a58eb154\") " pod="openshift-ingress/e2e-shared-a-w2-c165a4-openshift-default-5b5c6d994b-g4wbc" Sep 10 13:35:25.931690 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:35:25.931645 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b65e81d0-d6df-44d2-a87a-04b4a58eb154-workload-certs\") pod \"e2e-shared-a-w2-c165a4-openshift-default-5b5c6d994b-g4wbc\" (UID: \"b65e81d0-d6df-44d2-a87a-04b4a58eb154\") " pod="openshift-ingress/e2e-shared-a-w2-c165a4-openshift-default-5b5c6d994b-g4wbc" Sep 10 13:35:25.931835 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:35:25.931732 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b65e81d0-d6df-44d2-a87a-04b4a58eb154-istiod-ca-cert\") pod \"e2e-shared-a-w2-c165a4-openshift-default-5b5c6d994b-g4wbc\" (UID: \"b65e81d0-d6df-44d2-a87a-04b4a58eb154\") " pod="openshift-ingress/e2e-shared-a-w2-c165a4-openshift-default-5b5c6d994b-g4wbc" Sep 10 13:35:25.931835 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:35:25.931761 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b65e81d0-d6df-44d2-a87a-04b4a58eb154-credential-socket\") pod \"e2e-shared-a-w2-c165a4-openshift-default-5b5c6d994b-g4wbc\" (UID: \"b65e81d0-d6df-44d2-a87a-04b4a58eb154\") " pod="openshift-ingress/e2e-shared-a-w2-c165a4-openshift-default-5b5c6d994b-g4wbc" Sep 10 13:35:25.931960 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:35:25.931877 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b65e81d0-d6df-44d2-a87a-04b4a58eb154-istio-data\") pod \"e2e-shared-a-w2-c165a4-openshift-default-5b5c6d994b-g4wbc\" (UID: \"b65e81d0-d6df-44d2-a87a-04b4a58eb154\") " pod="openshift-ingress/e2e-shared-a-w2-c165a4-openshift-default-5b5c6d994b-g4wbc" Sep 10 13:35:25.933993 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:35:25.933972 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b65e81d0-d6df-44d2-a87a-04b4a58eb154-istio-podinfo\") pod \"e2e-shared-a-w2-c165a4-openshift-default-5b5c6d994b-g4wbc\" (UID: \"b65e81d0-d6df-44d2-a87a-04b4a58eb154\") " pod="openshift-ingress/e2e-shared-a-w2-c165a4-openshift-default-5b5c6d994b-g4wbc" Sep 10 13:35:25.934095 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:35:25.934056 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b65e81d0-d6df-44d2-a87a-04b4a58eb154-istio-envoy\") pod \"e2e-shared-a-w2-c165a4-openshift-default-5b5c6d994b-g4wbc\" (UID: \"b65e81d0-d6df-44d2-a87a-04b4a58eb154\") " pod="openshift-ingress/e2e-shared-a-w2-c165a4-openshift-default-5b5c6d994b-g4wbc" Sep 10 13:35:25.939793 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:35:25.939739 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2gvt2\" (UniqueName: \"kubernetes.io/projected/b65e81d0-d6df-44d2-a87a-04b4a58eb154-kube-api-access-2gvt2\") pod \"e2e-shared-a-w2-c165a4-openshift-default-5b5c6d994b-g4wbc\" (UID: \"b65e81d0-d6df-44d2-a87a-04b4a58eb154\") " pod="openshift-ingress/e2e-shared-a-w2-c165a4-openshift-default-5b5c6d994b-g4wbc" Sep 10 13:35:25.940235 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:35:25.940208 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b65e81d0-d6df-44d2-a87a-04b4a58eb154-istio-token\") pod \"e2e-shared-a-w2-c165a4-openshift-default-5b5c6d994b-g4wbc\" (UID: \"b65e81d0-d6df-44d2-a87a-04b4a58eb154\") " pod="openshift-ingress/e2e-shared-a-w2-c165a4-openshift-default-5b5c6d994b-g4wbc" Sep 10 13:35:26.010034 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:35:26.009985 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-c165a4-openshift-default-5b5c6d994b-g4wbc" Sep 10 13:35:26.304991 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:35:26.304957 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-c165a4-openshift-default-5b5c6d994b-g4wbc"] Sep 10 13:35:26.306491 ip-10-0-139-174 kubenswrapper[2557]: W0910 13:35:26.306455 2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb65e81d0_d6df_44d2_a87a_04b4a58eb154.slice/crio-87ce13f9691faf7d7e29c596867d46edb83cf1387e6fbaa6a516ffe4a55547c8 WatchSource:0}: Error finding container 87ce13f9691faf7d7e29c596867d46edb83cf1387e6fbaa6a516ffe4a55547c8: Status 404 returned error can't find the container with id 87ce13f9691faf7d7e29c596867d46edb83cf1387e6fbaa6a516ffe4a55547c8 Sep 10 13:35:26.309041 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:35:26.309003 2557 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892152Ki","pods":"250"} Sep 10 13:35:26.309134 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:35:26.309080 2557 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892152Ki","pods":"250"} Sep 10 13:35:26.309134 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:35:26.309123 2557 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892152Ki","pods":"250"} Sep 10 13:35:26.411610 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:35:26.411567 2557 patch_prober.go:28] interesting pod/e2e-shared-b-w3-7739cb-openshift-default-6764b957b8-zn8rv container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.65:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Sep 10 13:35:26.411800 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:35:26.411631 2557 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-b-w3-7739cb-openshift-default-6764b957b8-zn8rv" podUID="3b257e70-49de-40e6-940f-2306f4a81156" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.65:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Sep 10 13:35:26.412578 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:35:26.412556 2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-7739cb-79cc8c98c-gn8pb" Sep 10 13:35:26.412719 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:35:26.412627 2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-7739cb-5b465cd6d6-snwfr" Sep 10 13:35:26.497007 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:35:26.496970 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-c165a4-openshift-default-5b5c6d994b-g4wbc" event={"ID":"b65e81d0-d6df-44d2-a87a-04b4a58eb154","Type":"ContainerStarted","Data":"2cb49671b88e8282f7c5caa1f94efe48957c68fea70fb978a62b9c0445a20c56"} Sep 10 13:35:26.497174 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:35:26.497035 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-c165a4-openshift-default-5b5c6d994b-g4wbc" event={"ID":"b65e81d0-d6df-44d2-a87a-04b4a58eb154","Type":"ContainerStarted","Data":"87ce13f9691faf7d7e29c596867d46edb83cf1387e6fbaa6a516ffe4a55547c8"} Sep 10 13:35:26.520966 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:35:26.520872 2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w2-c165a4-openshift-default-5b5c6d994b-g4wbc" podStartSLOduration=1.520848365 podStartE2EDuration="1.520848365s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 13:35:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 13:35:26.52056182 +0000 UTC m=+1086.827556326" watchObservedRunningTime="2026-09-10 13:35:26.520848365 +0000 UTC m=+1086.827842780" Sep 10 13:35:27.010775 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:35:27.010750 2557 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w2-c165a4-openshift-default-5b5c6d994b-g4wbc" Sep 10 13:35:27.015609 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:35:27.015583 2557 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w2-c165a4-openshift-default-5b5c6d994b-g4wbc" Sep 10 13:35:27.503273 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:35:27.503240 2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w2-c165a4-openshift-default-5b5c6d994b-g4wbc" Sep 10 13:35:27.504336 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:35:27.504314 2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w2-c165a4-openshift-default-5b5c6d994b-g4wbc" Sep 10 13:35:28.510628 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:35:28.510541 2557 generic.go:356] "Generic (PLEG): container finished" podID="3b257e70-49de-40e6-940f-2306f4a81156" containerID="140c2c6a411ab954abe37e465cf3f295d26a01c3b2ae15ca0a7281a252ffcf9e" exitCode=0 Sep 10 13:35:28.510628 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:35:28.510614 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-7739cb-openshift-default-6764b957b8-zn8rv" event={"ID":"3b257e70-49de-40e6-940f-2306f4a81156","Type":"ContainerDied","Data":"140c2c6a411ab954abe37e465cf3f295d26a01c3b2ae15ca0a7281a252ffcf9e"} Sep 10 13:35:28.511147 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:35:28.511121 2557 scope.go:117] "RemoveContainer" containerID="140c2c6a411ab954abe37e465cf3f295d26a01c3b2ae15ca0a7281a252ffcf9e" Sep 10 13:35:28.512129 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:35:28.512104 2557 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892152Ki","pods":"250"} Sep 10 13:35:28.512180 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:35:28.512167 2557 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892152Ki","pods":"250"} Sep 10 13:35:28.512219 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:35:28.512197 2557 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892152Ki","pods":"250"} Sep 10 13:35:29.519011 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:35:29.518977 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-7739cb-openshift-default-6764b957b8-zn8rv" event={"ID":"3b257e70-49de-40e6-940f-2306f4a81156","Type":"ContainerStarted","Data":"f3ace53963b56f13c6cc784048663874a12d9c1583c4495fc4c216e8b44628f2"} Sep 10 13:35:29.597341 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:35:29.597308 2557 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w3-7739cb-openshift-default-6764b957b8-zn8rv" Sep 10 13:35:29.597569 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:35:29.597548 2557 patch_prober.go:28] interesting pod/e2e-shared-b-w3-7739cb-openshift-default-6764b957b8-zn8rv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.65:15021/healthz/ready\": dial tcp 10.134.0.65:15021: connect: connection refused" start-of-body= Sep 10 13:35:29.597633 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:35:29.597600 2557 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-7739cb-openshift-default-6764b957b8-zn8rv" podUID="3b257e70-49de-40e6-940f-2306f4a81156" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.65:15021/healthz/ready\": dial tcp 10.134.0.65:15021: connect: connection refused" Sep 10 13:35:30.597170 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:35:30.597133 2557 patch_prober.go:28] interesting pod/e2e-shared-b-w3-7739cb-openshift-default-6764b957b8-zn8rv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.65:15021/healthz/ready\": dial tcp 10.134.0.65:15021: connect: connection refused" start-of-body= Sep 10 13:35:30.597587 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:35:30.597194 2557 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-7739cb-openshift-default-6764b957b8-zn8rv" podUID="3b257e70-49de-40e6-940f-2306f4a81156" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.65:15021/healthz/ready\": dial tcp 10.134.0.65:15021: connect: connection refused" Sep 10 13:35:31.597548 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:35:31.597505 2557 patch_prober.go:28] interesting pod/e2e-shared-b-w3-7739cb-openshift-default-6764b957b8-zn8rv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.65:15021/healthz/ready\": dial tcp 10.134.0.65:15021: connect: connection refused" start-of-body= Sep 10 13:35:31.598023 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:35:31.597570 2557 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-7739cb-openshift-default-6764b957b8-zn8rv" podUID="3b257e70-49de-40e6-940f-2306f4a81156" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.65:15021/healthz/ready\": dial tcp 10.134.0.65:15021: connect: connection refused" Sep 10 13:35:32.597070 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:35:32.597037 2557 patch_prober.go:28] interesting pod/e2e-shared-b-w3-7739cb-openshift-default-6764b957b8-zn8rv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.65:15021/healthz/ready\": dial tcp 10.134.0.65:15021: connect: connection refused" start-of-body= Sep 10 13:35:32.597261 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:35:32.597091 2557 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-7739cb-openshift-default-6764b957b8-zn8rv" podUID="3b257e70-49de-40e6-940f-2306f4a81156" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.65:15021/healthz/ready\": dial tcp 10.134.0.65:15021: connect: connection refused" Sep 10 13:35:33.597493 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:35:33.597456 2557 patch_prober.go:28] interesting pod/e2e-shared-b-w3-7739cb-openshift-default-6764b957b8-zn8rv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.65:15021/healthz/ready\": dial tcp 10.134.0.65:15021: connect: connection refused" start-of-body= Sep 10 13:35:33.597878 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:35:33.597512 2557 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-7739cb-openshift-default-6764b957b8-zn8rv" podUID="3b257e70-49de-40e6-940f-2306f4a81156" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.65:15021/healthz/ready\": dial tcp 10.134.0.65:15021: connect: connection refused" Sep 10 13:35:34.631729 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:35:34.631695 2557 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w3-7739cb-openshift-default-6764b957b8-zn8rv" Sep 10 13:35:34.632696 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:35:34.632668 2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w3-7739cb-openshift-default-6764b957b8-zn8rv" Sep 10 13:35:34.633534 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:35:34.633509 2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w3-7739cb-openshift-default-6764b957b8-zn8rv" Sep 10 13:35:43.544678 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:35:43.544628 2557 patch_prober.go:28] interesting pod/e2e-shared-a-w2-c165a4-openshift-default-5b5c6d994b-g4wbc container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.68:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Sep 10 13:35:43.545139 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:35:43.544744 2557 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-a-w2-c165a4-openshift-default-5b5c6d994b-g4wbc" podUID="b65e81d0-d6df-44d2-a87a-04b4a58eb154" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.68:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Sep 10 13:35:45.625004 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:35:45.624960 2557 generic.go:356] "Generic (PLEG): container finished" podID="b65e81d0-d6df-44d2-a87a-04b4a58eb154" containerID="2cb49671b88e8282f7c5caa1f94efe48957c68fea70fb978a62b9c0445a20c56" exitCode=0 Sep 10 13:35:45.625528 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:35:45.625039 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-c165a4-openshift-default-5b5c6d994b-g4wbc" event={"ID":"b65e81d0-d6df-44d2-a87a-04b4a58eb154","Type":"ContainerDied","Data":"2cb49671b88e8282f7c5caa1f94efe48957c68fea70fb978a62b9c0445a20c56"} Sep 10 13:35:45.625601 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:35:45.625578 2557 scope.go:117] "RemoveContainer" containerID="2cb49671b88e8282f7c5caa1f94efe48957c68fea70fb978a62b9c0445a20c56" Sep 10 13:35:45.626947 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:35:45.626913 2557 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892152Ki","pods":"250"} Sep 10 13:35:45.627089 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:35:45.626994 2557 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892152Ki","pods":"250"} Sep 10 13:35:45.627089 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:35:45.627029 2557 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892152Ki","pods":"250"} Sep 10 13:35:46.632677 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:35:46.632642 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-c165a4-openshift-default-5b5c6d994b-g4wbc" event={"ID":"b65e81d0-d6df-44d2-a87a-04b4a58eb154","Type":"ContainerStarted","Data":"c1553bb4051824e55bf4fb30a37e64d5f2f738c8ced98dbc30e6ce76c2251d75"} Sep 10 13:35:47.010945 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:35:47.010909 2557 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w2-c165a4-openshift-default-5b5c6d994b-g4wbc" Sep 10 13:35:47.011164 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:35:47.011144 2557 patch_prober.go:28] interesting pod/e2e-shared-a-w2-c165a4-openshift-default-5b5c6d994b-g4wbc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.68:15021/healthz/ready\": dial tcp 10.134.0.68:15021: connect: connection refused" start-of-body= Sep 10 13:35:47.011214 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:35:47.011199 2557 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-c165a4-openshift-default-5b5c6d994b-g4wbc" podUID="b65e81d0-d6df-44d2-a87a-04b4a58eb154" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.68:15021/healthz/ready\": dial tcp 10.134.0.68:15021: connect: connection refused" Sep 10 13:35:48.010712 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:35:48.010676 2557 patch_prober.go:28] interesting pod/e2e-shared-a-w2-c165a4-openshift-default-5b5c6d994b-g4wbc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.68:15021/healthz/ready\": dial tcp 10.134.0.68:15021: connect: connection refused" start-of-body= Sep 10 13:35:48.011136 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:35:48.010730 2557 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-c165a4-openshift-default-5b5c6d994b-g4wbc" podUID="b65e81d0-d6df-44d2-a87a-04b4a58eb154" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.68:15021/healthz/ready\": dial tcp 10.134.0.68:15021: connect: connection refused" Sep 10 13:35:49.010749 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:35:49.010708 2557 patch_prober.go:28] interesting pod/e2e-shared-a-w2-c165a4-openshift-default-5b5c6d994b-g4wbc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.68:15021/healthz/ready\": dial tcp 10.134.0.68:15021: connect: connection refused" start-of-body= Sep 10 13:35:49.011248 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:35:49.010772 2557 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-c165a4-openshift-default-5b5c6d994b-g4wbc" podUID="b65e81d0-d6df-44d2-a87a-04b4a58eb154" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.68:15021/healthz/ready\": dial tcp 10.134.0.68:15021: connect: connection refused" Sep 10 13:35:50.010551 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:35:50.010508 2557 patch_prober.go:28] interesting pod/e2e-shared-a-w2-c165a4-openshift-default-5b5c6d994b-g4wbc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.68:15021/healthz/ready\": dial tcp 10.134.0.68:15021: connect: connection refused" start-of-body= Sep 10 13:35:50.010778 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:35:50.010570 2557 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-c165a4-openshift-default-5b5c6d994b-g4wbc" podUID="b65e81d0-d6df-44d2-a87a-04b4a58eb154" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.68:15021/healthz/ready\": dial tcp 10.134.0.68:15021: connect: connection refused" Sep 10 13:35:51.010635 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:35:51.010600 2557 patch_prober.go:28] interesting pod/e2e-shared-a-w2-c165a4-openshift-default-5b5c6d994b-g4wbc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.68:15021/healthz/ready\": dial tcp 10.134.0.68:15021: connect: connection refused" start-of-body= Sep 10 13:35:51.010809 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:35:51.010663 2557 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-c165a4-openshift-default-5b5c6d994b-g4wbc" podUID="b65e81d0-d6df-44d2-a87a-04b4a58eb154" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.68:15021/healthz/ready\": dial tcp 10.134.0.68:15021: connect: connection refused" Sep 10 13:35:52.206933 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:35:52.206534 2557 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w2-c165a4-openshift-default-5b5c6d994b-g4wbc" Sep 10 13:35:52.207449 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:35:52.206983 2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w2-c165a4-openshift-default-5b5c6d994b-g4wbc" Sep 10 13:35:52.208069 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:35:52.208045 2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w2-c165a4-openshift-default-5b5c6d994b-g4wbc" Sep 10 13:35:54.349046 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:35:54.349014 2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w2-e65037-openshift-default-5c9697d44-nm2bl"] Sep 10 13:35:54.352086 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:35:54.352068 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-e65037-openshift-default-5c9697d44-nm2bl" Sep 10 13:35:54.367492 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:35:54.367472 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w2-e65037-openshift-default-dockercfg-4zvnt\"" Sep 10 13:35:54.426993 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:35:54.426964 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-e65037-openshift-default-5c9697d44-nm2bl"] Sep 10 13:35:54.533376 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:35:54.533337 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9c46315e-65c5-42f5-81de-91e639e6ed98-istio-data\") pod \"e2e-shared-b-w2-e65037-openshift-default-5c9697d44-nm2bl\" (UID: \"9c46315e-65c5-42f5-81de-91e639e6ed98\") " pod="openshift-ingress/e2e-shared-b-w2-e65037-openshift-default-5c9697d44-nm2bl" Sep 10 13:35:54.533565 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:35:54.533388 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9c46315e-65c5-42f5-81de-91e639e6ed98-istio-token\") pod \"e2e-shared-b-w2-e65037-openshift-default-5c9697d44-nm2bl\" (UID: \"9c46315e-65c5-42f5-81de-91e639e6ed98\") " pod="openshift-ingress/e2e-shared-b-w2-e65037-openshift-default-5c9697d44-nm2bl" Sep 10 13:35:54.533565 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:35:54.533442 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9c46315e-65c5-42f5-81de-91e639e6ed98-credential-socket\") pod \"e2e-shared-b-w2-e65037-openshift-default-5c9697d44-nm2bl\" (UID: \"9c46315e-65c5-42f5-81de-91e639e6ed98\") " pod="openshift-ingress/e2e-shared-b-w2-e65037-openshift-default-5c9697d44-nm2bl" Sep 10 13:35:54.533565 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:35:54.533459 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9c46315e-65c5-42f5-81de-91e639e6ed98-istio-envoy\") pod \"e2e-shared-b-w2-e65037-openshift-default-5c9697d44-nm2bl\" (UID: \"9c46315e-65c5-42f5-81de-91e639e6ed98\") " pod="openshift-ingress/e2e-shared-b-w2-e65037-openshift-default-5c9697d44-nm2bl" Sep 10 13:35:54.533565 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:35:54.533477 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7r7d7\" (UniqueName: \"kubernetes.io/projected/9c46315e-65c5-42f5-81de-91e639e6ed98-kube-api-access-7r7d7\") pod \"e2e-shared-b-w2-e65037-openshift-default-5c9697d44-nm2bl\" (UID: \"9c46315e-65c5-42f5-81de-91e639e6ed98\") " pod="openshift-ingress/e2e-shared-b-w2-e65037-openshift-default-5c9697d44-nm2bl" Sep 10 13:35:54.533565 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:35:54.533555 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9c46315e-65c5-42f5-81de-91e639e6ed98-istio-podinfo\") pod \"e2e-shared-b-w2-e65037-openshift-default-5c9697d44-nm2bl\" (UID: \"9c46315e-65c5-42f5-81de-91e639e6ed98\") " pod="openshift-ingress/e2e-shared-b-w2-e65037-openshift-default-5c9697d44-nm2bl" Sep 10 13:35:54.533739 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:35:54.533595 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9c46315e-65c5-42f5-81de-91e639e6ed98-istiod-ca-cert\") pod \"e2e-shared-b-w2-e65037-openshift-default-5c9697d44-nm2bl\" (UID: \"9c46315e-65c5-42f5-81de-91e639e6ed98\") " pod="openshift-ingress/e2e-shared-b-w2-e65037-openshift-default-5c9697d44-nm2bl" Sep 10 13:35:54.533739 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:35:54.533660 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9c46315e-65c5-42f5-81de-91e639e6ed98-workload-socket\") pod \"e2e-shared-b-w2-e65037-openshift-default-5c9697d44-nm2bl\" (UID: \"9c46315e-65c5-42f5-81de-91e639e6ed98\") " pod="openshift-ingress/e2e-shared-b-w2-e65037-openshift-default-5c9697d44-nm2bl" Sep 10 13:35:54.533739 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:35:54.533692 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9c46315e-65c5-42f5-81de-91e639e6ed98-workload-certs\") pod \"e2e-shared-b-w2-e65037-openshift-default-5c9697d44-nm2bl\" (UID: \"9c46315e-65c5-42f5-81de-91e639e6ed98\") " pod="openshift-ingress/e2e-shared-b-w2-e65037-openshift-default-5c9697d44-nm2bl" Sep 10 13:35:54.635145 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:35:54.635077 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9c46315e-65c5-42f5-81de-91e639e6ed98-istio-data\") pod \"e2e-shared-b-w2-e65037-openshift-default-5c9697d44-nm2bl\" (UID: \"9c46315e-65c5-42f5-81de-91e639e6ed98\") " pod="openshift-ingress/e2e-shared-b-w2-e65037-openshift-default-5c9697d44-nm2bl" Sep 10 13:35:54.635145 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:35:54.635121 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9c46315e-65c5-42f5-81de-91e639e6ed98-istio-token\") pod \"e2e-shared-b-w2-e65037-openshift-default-5c9697d44-nm2bl\" (UID: \"9c46315e-65c5-42f5-81de-91e639e6ed98\") " pod="openshift-ingress/e2e-shared-b-w2-e65037-openshift-default-5c9697d44-nm2bl" Sep 10 13:35:54.635145 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:35:54.635145 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9c46315e-65c5-42f5-81de-91e639e6ed98-credential-socket\") pod \"e2e-shared-b-w2-e65037-openshift-default-5c9697d44-nm2bl\" (UID: \"9c46315e-65c5-42f5-81de-91e639e6ed98\") " pod="openshift-ingress/e2e-shared-b-w2-e65037-openshift-default-5c9697d44-nm2bl" Sep 10 13:35:54.635413 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:35:54.635159 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9c46315e-65c5-42f5-81de-91e639e6ed98-istio-envoy\") pod \"e2e-shared-b-w2-e65037-openshift-default-5c9697d44-nm2bl\" (UID: \"9c46315e-65c5-42f5-81de-91e639e6ed98\") " pod="openshift-ingress/e2e-shared-b-w2-e65037-openshift-default-5c9697d44-nm2bl" Sep 10 13:35:54.635413 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:35:54.635176 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7r7d7\" (UniqueName: \"kubernetes.io/projected/9c46315e-65c5-42f5-81de-91e639e6ed98-kube-api-access-7r7d7\") pod \"e2e-shared-b-w2-e65037-openshift-default-5c9697d44-nm2bl\" (UID: \"9c46315e-65c5-42f5-81de-91e639e6ed98\") " pod="openshift-ingress/e2e-shared-b-w2-e65037-openshift-default-5c9697d44-nm2bl" Sep 10 13:35:54.635413 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:35:54.635196 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9c46315e-65c5-42f5-81de-91e639e6ed98-istio-podinfo\") pod \"e2e-shared-b-w2-e65037-openshift-default-5c9697d44-nm2bl\" (UID: \"9c46315e-65c5-42f5-81de-91e639e6ed98\") " pod="openshift-ingress/e2e-shared-b-w2-e65037-openshift-default-5c9697d44-nm2bl" Sep 10 13:35:54.635413 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:35:54.635219 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9c46315e-65c5-42f5-81de-91e639e6ed98-istiod-ca-cert\") pod \"e2e-shared-b-w2-e65037-openshift-default-5c9697d44-nm2bl\" (UID: \"9c46315e-65c5-42f5-81de-91e639e6ed98\") " pod="openshift-ingress/e2e-shared-b-w2-e65037-openshift-default-5c9697d44-nm2bl" Sep 10 13:35:54.635413 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:35:54.635367 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9c46315e-65c5-42f5-81de-91e639e6ed98-workload-socket\") pod \"e2e-shared-b-w2-e65037-openshift-default-5c9697d44-nm2bl\" (UID: \"9c46315e-65c5-42f5-81de-91e639e6ed98\") " pod="openshift-ingress/e2e-shared-b-w2-e65037-openshift-default-5c9697d44-nm2bl" Sep 10 13:35:54.635413 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:35:54.635409 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9c46315e-65c5-42f5-81de-91e639e6ed98-workload-certs\") pod \"e2e-shared-b-w2-e65037-openshift-default-5c9697d44-nm2bl\" (UID: \"9c46315e-65c5-42f5-81de-91e639e6ed98\") " pod="openshift-ingress/e2e-shared-b-w2-e65037-openshift-default-5c9697d44-nm2bl" Sep 10 13:35:54.635788 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:35:54.635513 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9c46315e-65c5-42f5-81de-91e639e6ed98-istio-data\") pod \"e2e-shared-b-w2-e65037-openshift-default-5c9697d44-nm2bl\" (UID: \"9c46315e-65c5-42f5-81de-91e639e6ed98\") " pod="openshift-ingress/e2e-shared-b-w2-e65037-openshift-default-5c9697d44-nm2bl" Sep 10 13:35:54.635788 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:35:54.635557 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9c46315e-65c5-42f5-81de-91e639e6ed98-credential-socket\") pod \"e2e-shared-b-w2-e65037-openshift-default-5c9697d44-nm2bl\" (UID: \"9c46315e-65c5-42f5-81de-91e639e6ed98\") " pod="openshift-ingress/e2e-shared-b-w2-e65037-openshift-default-5c9697d44-nm2bl" Sep 10 13:35:54.635788 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:35:54.635733 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9c46315e-65c5-42f5-81de-91e639e6ed98-workload-socket\") pod \"e2e-shared-b-w2-e65037-openshift-default-5c9697d44-nm2bl\" (UID: \"9c46315e-65c5-42f5-81de-91e639e6ed98\") " pod="openshift-ingress/e2e-shared-b-w2-e65037-openshift-default-5c9697d44-nm2bl" Sep 10 13:35:54.635788 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:35:54.635777 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9c46315e-65c5-42f5-81de-91e639e6ed98-workload-certs\") pod \"e2e-shared-b-w2-e65037-openshift-default-5c9697d44-nm2bl\" (UID: \"9c46315e-65c5-42f5-81de-91e639e6ed98\") " pod="openshift-ingress/e2e-shared-b-w2-e65037-openshift-default-5c9697d44-nm2bl" Sep 10 13:35:54.636041 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:35:54.636023 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9c46315e-65c5-42f5-81de-91e639e6ed98-istiod-ca-cert\") pod \"e2e-shared-b-w2-e65037-openshift-default-5c9697d44-nm2bl\" (UID: \"9c46315e-65c5-42f5-81de-91e639e6ed98\") " pod="openshift-ingress/e2e-shared-b-w2-e65037-openshift-default-5c9697d44-nm2bl" Sep 10 13:35:54.637749 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:35:54.637722 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9c46315e-65c5-42f5-81de-91e639e6ed98-istio-envoy\") pod \"e2e-shared-b-w2-e65037-openshift-default-5c9697d44-nm2bl\" (UID: \"9c46315e-65c5-42f5-81de-91e639e6ed98\") " pod="openshift-ingress/e2e-shared-b-w2-e65037-openshift-default-5c9697d44-nm2bl" Sep 10 13:35:54.637849 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:35:54.637769 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9c46315e-65c5-42f5-81de-91e639e6ed98-istio-podinfo\") pod \"e2e-shared-b-w2-e65037-openshift-default-5c9697d44-nm2bl\" (UID: \"9c46315e-65c5-42f5-81de-91e639e6ed98\") " pod="openshift-ingress/e2e-shared-b-w2-e65037-openshift-default-5c9697d44-nm2bl" Sep 10 13:35:54.645240 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:35:54.645222 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9c46315e-65c5-42f5-81de-91e639e6ed98-istio-token\") pod \"e2e-shared-b-w2-e65037-openshift-default-5c9697d44-nm2bl\" (UID: \"9c46315e-65c5-42f5-81de-91e639e6ed98\") " pod="openshift-ingress/e2e-shared-b-w2-e65037-openshift-default-5c9697d44-nm2bl" Sep 10 13:35:54.645588 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:35:54.645569 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7r7d7\" (UniqueName: \"kubernetes.io/projected/9c46315e-65c5-42f5-81de-91e639e6ed98-kube-api-access-7r7d7\") pod \"e2e-shared-b-w2-e65037-openshift-default-5c9697d44-nm2bl\" (UID: \"9c46315e-65c5-42f5-81de-91e639e6ed98\") " pod="openshift-ingress/e2e-shared-b-w2-e65037-openshift-default-5c9697d44-nm2bl" Sep 10 13:35:54.662307 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:35:54.662290 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-e65037-openshift-default-5c9697d44-nm2bl" Sep 10 13:35:54.948085 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:35:54.948061 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-e65037-openshift-default-5c9697d44-nm2bl"] Sep 10 13:35:54.949447 ip-10-0-139-174 kubenswrapper[2557]: W0910 13:35:54.949397 2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9c46315e_65c5_42f5_81de_91e639e6ed98.slice/crio-4163482345bd5cdc6443e7e5acf67554dfb272a479cb14005088143778d958e4 WatchSource:0}: Error finding container 4163482345bd5cdc6443e7e5acf67554dfb272a479cb14005088143778d958e4: Status 404 returned error can't find the container with id 4163482345bd5cdc6443e7e5acf67554dfb272a479cb14005088143778d958e4 Sep 10 13:35:54.951924 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:35:54.951887 2557 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892152Ki","pods":"250"} Sep 10 13:35:54.952015 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:35:54.951961 2557 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892152Ki","pods":"250"} Sep 10 13:35:54.952015 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:35:54.951990 2557 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892152Ki","pods":"250"} Sep 10 13:35:55.700707 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:35:55.700675 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-e65037-openshift-default-5c9697d44-nm2bl" event={"ID":"9c46315e-65c5-42f5-81de-91e639e6ed98","Type":"ContainerStarted","Data":"e2175dab6789a058f8364fb79878fe74a180e2ad85a198d39c28f96d199e7415"} Sep 10 13:35:55.701066 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:35:55.700722 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-e65037-openshift-default-5c9697d44-nm2bl" event={"ID":"9c46315e-65c5-42f5-81de-91e639e6ed98","Type":"ContainerStarted","Data":"4163482345bd5cdc6443e7e5acf67554dfb272a479cb14005088143778d958e4"} Sep 10 13:35:55.737969 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:35:55.737918 2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w2-e65037-openshift-default-5c9697d44-nm2bl" podStartSLOduration=1.737897926 podStartE2EDuration="1.737897926s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 13:35:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 13:35:55.735889698 +0000 UTC m=+1116.042884114" watchObservedRunningTime="2026-09-10 13:35:55.737897926 +0000 UTC m=+1116.044892339" Sep 10 13:35:56.662870 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:35:56.662832 2557 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w2-e65037-openshift-default-5c9697d44-nm2bl" Sep 10 13:35:56.668551 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:35:56.668525 2557 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w2-e65037-openshift-default-5c9697d44-nm2bl" Sep 10 13:35:56.706589 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:35:56.706564 2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w2-e65037-openshift-default-5c9697d44-nm2bl" Sep 10 13:35:56.707471 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:35:56.707452 2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w2-e65037-openshift-default-5c9697d44-nm2bl" Sep 10 13:35:59.945053 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:35:59.945016 2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-e65037-7c58dd8dff-g4wd6"] Sep 10 13:35:59.949737 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:35:59.949713 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-e65037-7c58dd8dff-g4wd6" Sep 10 13:35:59.954800 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:35:59.954771 2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-w2-e65037\"" Sep 10 13:35:59.955493 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:35:59.955088 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-b-w2-e65037-dockercfg-fnh7w\"" Sep 10 13:35:59.968335 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:35:59.968305 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-e65037-7c58dd8dff-g4wd6"] Sep 10 13:36:00.088836 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:36:00.088801 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b8drm\" (UniqueName: \"kubernetes.io/projected/1d6c6e09-c18e-4388-a708-1ad80cd647ed-kube-api-access-b8drm\") pod \"payload-processing-e2e-shared-b-w2-e65037-7c58dd8dff-g4wd6\" (UID: \"1d6c6e09-c18e-4388-a708-1ad80cd647ed\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-e65037-7c58dd8dff-g4wd6" Sep 10 13:36:00.088985 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:36:00.088903 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1d6c6e09-c18e-4388-a708-1ad80cd647ed-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w2-e65037-7c58dd8dff-g4wd6\" (UID: \"1d6c6e09-c18e-4388-a708-1ad80cd647ed\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-e65037-7c58dd8dff-g4wd6" Sep 10 13:36:00.190229 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:36:00.190192 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b8drm\" (UniqueName: \"kubernetes.io/projected/1d6c6e09-c18e-4388-a708-1ad80cd647ed-kube-api-access-b8drm\") pod \"payload-processing-e2e-shared-b-w2-e65037-7c58dd8dff-g4wd6\" (UID: \"1d6c6e09-c18e-4388-a708-1ad80cd647ed\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-e65037-7c58dd8dff-g4wd6" Sep 10 13:36:00.190382 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:36:00.190282 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1d6c6e09-c18e-4388-a708-1ad80cd647ed-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w2-e65037-7c58dd8dff-g4wd6\" (UID: \"1d6c6e09-c18e-4388-a708-1ad80cd647ed\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-e65037-7c58dd8dff-g4wd6" Sep 10 13:36:00.191112 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:36:00.191078 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1d6c6e09-c18e-4388-a708-1ad80cd647ed-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w2-e65037-7c58dd8dff-g4wd6\" (UID: \"1d6c6e09-c18e-4388-a708-1ad80cd647ed\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-e65037-7c58dd8dff-g4wd6" Sep 10 13:36:00.202090 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:36:00.202034 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-b8drm\" (UniqueName: \"kubernetes.io/projected/1d6c6e09-c18e-4388-a708-1ad80cd647ed-kube-api-access-b8drm\") pod \"payload-processing-e2e-shared-b-w2-e65037-7c58dd8dff-g4wd6\" (UID: \"1d6c6e09-c18e-4388-a708-1ad80cd647ed\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-e65037-7c58dd8dff-g4wd6" Sep 10 13:36:00.271561 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:36:00.271539 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-e65037-7c58dd8dff-g4wd6" Sep 10 13:36:00.550075 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:36:00.550043 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-e65037-7c58dd8dff-g4wd6"] Sep 10 13:36:00.550517 ip-10-0-139-174 kubenswrapper[2557]: W0910 13:36:00.550494 2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1d6c6e09_c18e_4388_a708_1ad80cd647ed.slice/crio-873ef8831b7aac12d2eb0056ae796a29dd29db94bdd9ba6f56ee0227d30d76a4 WatchSource:0}: Error finding container 873ef8831b7aac12d2eb0056ae796a29dd29db94bdd9ba6f56ee0227d30d76a4: Status 404 returned error can't find the container with id 873ef8831b7aac12d2eb0056ae796a29dd29db94bdd9ba6f56ee0227d30d76a4 Sep 10 13:36:00.734388 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:36:00.734354 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-e65037-7c58dd8dff-g4wd6" event={"ID":"1d6c6e09-c18e-4388-a708-1ad80cd647ed","Type":"ContainerStarted","Data":"75110f0dcd52679d4559ab20d3f782320435abe3bf7d6ba734cb669c179835ee"} Sep 10 13:36:00.734590 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:36:00.734415 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-e65037-7c58dd8dff-g4wd6" event={"ID":"1d6c6e09-c18e-4388-a708-1ad80cd647ed","Type":"ContainerStarted","Data":"873ef8831b7aac12d2eb0056ae796a29dd29db94bdd9ba6f56ee0227d30d76a4"} Sep 10 13:36:00.734590 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:36:00.734562 2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-e65037-7c58dd8dff-g4wd6" Sep 10 13:36:00.755655 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:36:00.755613 2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-e65037-7c58dd8dff-g4wd6" podStartSLOduration=1.755595937 podStartE2EDuration="1.755595937s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 13:35:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 13:36:00.753848445 +0000 UTC m=+1121.060842860" watchObservedRunningTime="2026-09-10 13:36:00.755595937 +0000 UTC m=+1121.062590354" Sep 10 13:36:11.743613 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:36:11.743581 2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-e65037-7c58dd8dff-g4wd6" Sep 10 13:36:16.838266 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:36:16.838188 2557 generic.go:356] "Generic (PLEG): container finished" podID="9c46315e-65c5-42f5-81de-91e639e6ed98" containerID="e2175dab6789a058f8364fb79878fe74a180e2ad85a198d39c28f96d199e7415" exitCode=0 Sep 10 13:36:16.838661 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:36:16.838268 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-e65037-openshift-default-5c9697d44-nm2bl" event={"ID":"9c46315e-65c5-42f5-81de-91e639e6ed98","Type":"ContainerDied","Data":"e2175dab6789a058f8364fb79878fe74a180e2ad85a198d39c28f96d199e7415"} Sep 10 13:36:16.838713 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:36:16.838698 2557 scope.go:117] "RemoveContainer" containerID="e2175dab6789a058f8364fb79878fe74a180e2ad85a198d39c28f96d199e7415" Sep 10 13:36:16.839758 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:36:16.839730 2557 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892152Ki","pods":"250"} Sep 10 13:36:16.839809 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:36:16.839799 2557 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892152Ki","pods":"250"} Sep 10 13:36:16.839857 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:36:16.839839 2557 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892152Ki","pods":"250"} Sep 10 13:36:17.849710 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:36:17.849663 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-e65037-openshift-default-5c9697d44-nm2bl" event={"ID":"9c46315e-65c5-42f5-81de-91e639e6ed98","Type":"ContainerStarted","Data":"80764c4fc5dde5bd235d79f385ff53176e37622fb3633ef124968ab1ccbc9666"} Sep 10 13:36:18.663057 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:36:18.663024 2557 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w2-e65037-openshift-default-5c9697d44-nm2bl" Sep 10 13:36:18.663713 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:36:18.663272 2557 patch_prober.go:28] interesting pod/e2e-shared-b-w2-e65037-openshift-default-5c9697d44-nm2bl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.69:15021/healthz/ready\": dial tcp 10.134.0.69:15021: connect: connection refused" start-of-body= Sep 10 13:36:18.663713 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:36:18.663439 2557 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-e65037-openshift-default-5c9697d44-nm2bl" podUID="9c46315e-65c5-42f5-81de-91e639e6ed98" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.69:15021/healthz/ready\": dial tcp 10.134.0.69:15021: connect: connection refused" Sep 10 13:36:19.663761 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:36:19.663722 2557 patch_prober.go:28] interesting pod/e2e-shared-b-w2-e65037-openshift-default-5c9697d44-nm2bl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.69:15021/healthz/ready\": dial tcp 10.134.0.69:15021: connect: connection refused" start-of-body= Sep 10 13:36:19.664370 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:36:19.664338 2557 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-e65037-openshift-default-5c9697d44-nm2bl" podUID="9c46315e-65c5-42f5-81de-91e639e6ed98" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.69:15021/healthz/ready\": dial tcp 10.134.0.69:15021: connect: connection refused" Sep 10 13:36:20.299714 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:36:20.299681 2557 scope.go:117] "RemoveContainer" containerID="bea50f533924e113fb4d6777c58e74f6f2d1371693833998bfb29b2bc9f387eb" Sep 10 13:36:20.308872 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:36:20.308848 2557 scope.go:117] "RemoveContainer" containerID="2ec382f4e6e076dd64cdbf0404509c3767f7225af9f9287ba2d9ed797574fca7" Sep 10 13:36:20.663171 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:36:20.663065 2557 patch_prober.go:28] interesting pod/e2e-shared-b-w2-e65037-openshift-default-5c9697d44-nm2bl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.69:15021/healthz/ready\": dial tcp 10.134.0.69:15021: connect: connection refused" start-of-body= Sep 10 13:36:20.663171 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:36:20.663137 2557 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-e65037-openshift-default-5c9697d44-nm2bl" podUID="9c46315e-65c5-42f5-81de-91e639e6ed98" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.69:15021/healthz/ready\": dial tcp 10.134.0.69:15021: connect: connection refused" Sep 10 13:36:21.663620 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:36:21.663148 2557 patch_prober.go:28] interesting pod/e2e-shared-b-w2-e65037-openshift-default-5c9697d44-nm2bl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.69:15021/healthz/ready\": dial tcp 10.134.0.69:15021: connect: connection refused" start-of-body= Sep 10 13:36:21.663620 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:36:21.663213 2557 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-e65037-openshift-default-5c9697d44-nm2bl" podUID="9c46315e-65c5-42f5-81de-91e639e6ed98" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.69:15021/healthz/ready\": dial tcp 10.134.0.69:15021: connect: connection refused" Sep 10 13:36:22.663166 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:36:22.663131 2557 patch_prober.go:28] interesting pod/e2e-shared-b-w2-e65037-openshift-default-5c9697d44-nm2bl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.69:15021/healthz/ready\": dial tcp 10.134.0.69:15021: connect: connection refused" start-of-body= Sep 10 13:36:22.663474 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:36:22.663443 2557 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-e65037-openshift-default-5c9697d44-nm2bl" podUID="9c46315e-65c5-42f5-81de-91e639e6ed98" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.69:15021/healthz/ready\": dial tcp 10.134.0.69:15021: connect: connection refused" Sep 10 13:36:23.667396 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:36:23.667366 2557 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w2-e65037-openshift-default-5c9697d44-nm2bl" Sep 10 13:36:23.667841 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:36:23.667663 2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w2-e65037-openshift-default-5c9697d44-nm2bl" Sep 10 13:36:23.668342 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:36:23.668328 2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w2-e65037-openshift-default-5c9697d44-nm2bl" Sep 10 13:36:40.018863 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:36:40.018823 2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-3ca6fbf6-5b467d68c4-g7b2q"] Sep 10 13:36:40.022361 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:36:40.022345 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-3ca6fbf6-5b467d68c4-g7b2q" Sep 10 13:36:40.024632 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:36:40.024608 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-mt-3ca6fbf6-dockercfg-g6cbv\"" Sep 10 13:36:40.024745 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:36:40.024610 2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-mt-3ca6fbf6\"" Sep 10 13:36:40.032802 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:36:40.032784 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-3ca6fbf6-5b467d68c4-g7b2q"] Sep 10 13:36:40.179300 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:36:40.179264 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8778a756-cdb8-4a9c-be8c-ef1a8ad373bc-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-3ca6fbf6-5b467d68c4-g7b2q\" (UID: \"8778a756-cdb8-4a9c-be8c-ef1a8ad373bc\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-3ca6fbf6-5b467d68c4-g7b2q" Sep 10 13:36:40.179300 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:36:40.179299 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bpdxg\" (UniqueName: \"kubernetes.io/projected/8778a756-cdb8-4a9c-be8c-ef1a8ad373bc-kube-api-access-bpdxg\") pod \"payload-pre-processing-e2e-mt-3ca6fbf6-5b467d68c4-g7b2q\" (UID: \"8778a756-cdb8-4a9c-be8c-ef1a8ad373bc\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-3ca6fbf6-5b467d68c4-g7b2q" Sep 10 13:36:40.280336 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:36:40.280252 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8778a756-cdb8-4a9c-be8c-ef1a8ad373bc-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-3ca6fbf6-5b467d68c4-g7b2q\" (UID: \"8778a756-cdb8-4a9c-be8c-ef1a8ad373bc\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-3ca6fbf6-5b467d68c4-g7b2q" Sep 10 13:36:40.280336 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:36:40.280285 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bpdxg\" (UniqueName: \"kubernetes.io/projected/8778a756-cdb8-4a9c-be8c-ef1a8ad373bc-kube-api-access-bpdxg\") pod \"payload-pre-processing-e2e-mt-3ca6fbf6-5b467d68c4-g7b2q\" (UID: \"8778a756-cdb8-4a9c-be8c-ef1a8ad373bc\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-3ca6fbf6-5b467d68c4-g7b2q" Sep 10 13:36:40.280980 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:36:40.280956 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8778a756-cdb8-4a9c-be8c-ef1a8ad373bc-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-3ca6fbf6-5b467d68c4-g7b2q\" (UID: \"8778a756-cdb8-4a9c-be8c-ef1a8ad373bc\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-3ca6fbf6-5b467d68c4-g7b2q" Sep 10 13:36:40.289083 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:36:40.289055 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bpdxg\" (UniqueName: \"kubernetes.io/projected/8778a756-cdb8-4a9c-be8c-ef1a8ad373bc-kube-api-access-bpdxg\") pod \"payload-pre-processing-e2e-mt-3ca6fbf6-5b467d68c4-g7b2q\" (UID: \"8778a756-cdb8-4a9c-be8c-ef1a8ad373bc\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-3ca6fbf6-5b467d68c4-g7b2q" Sep 10 13:36:40.334045 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:36:40.334015 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-3ca6fbf6-5b467d68c4-g7b2q" Sep 10 13:36:40.606258 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:36:40.606232 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-3ca6fbf6-5b467d68c4-g7b2q"] Sep 10 13:36:40.607723 ip-10-0-139-174 kubenswrapper[2557]: W0910 13:36:40.607694 2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8778a756_cdb8_4a9c_be8c_ef1a8ad373bc.slice/crio-68c1b22418064056030f24ecd03ce4739e130a59f673c9a4305d16e177ff10f0 WatchSource:0}: Error finding container 68c1b22418064056030f24ecd03ce4739e130a59f673c9a4305d16e177ff10f0: Status 404 returned error can't find the container with id 68c1b22418064056030f24ecd03ce4739e130a59f673c9a4305d16e177ff10f0 Sep 10 13:36:41.001372 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:36:41.001337 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-3ca6fbf6-5b467d68c4-g7b2q" event={"ID":"8778a756-cdb8-4a9c-be8c-ef1a8ad373bc","Type":"ContainerStarted","Data":"b0afe0273015a68e9be39758483965bcb971e767382a3d58f76a4f46a0572661"} Sep 10 13:36:41.001583 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:36:41.001384 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-3ca6fbf6-5b467d68c4-g7b2q" event={"ID":"8778a756-cdb8-4a9c-be8c-ef1a8ad373bc","Type":"ContainerStarted","Data":"68c1b22418064056030f24ecd03ce4739e130a59f673c9a4305d16e177ff10f0"} Sep 10 13:36:41.001583 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:36:41.001457 2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-3ca6fbf6-5b467d68c4-g7b2q" Sep 10 13:36:41.020472 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:36:41.020403 2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-mt-3ca6fbf6-5b467d68c4-g7b2q" podStartSLOduration=1.020390174 podStartE2EDuration="1.020390174s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 13:36:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 13:36:41.01962812 +0000 UTC m=+1161.326622532" watchObservedRunningTime="2026-09-10 13:36:41.020390174 +0000 UTC m=+1161.327384587" Sep 10 13:36:52.009430 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:36:52.009396 2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-3ca6fbf6-5b467d68c4-g7b2q" Sep 10 13:37:20.227214 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:37:20.227185 2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-92n4q_720acda6-0b15-49ac-ab4e-60de70c4324a/ovn-acl-logging/0.log" Sep 10 13:37:20.231816 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:37:20.231795 2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-92n4q_720acda6-0b15-49ac-ab4e-60de70c4324a/ovn-acl-logging/0.log" Sep 10 13:37:20.337108 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:37:20.337077 2557 scope.go:117] "RemoveContainer" containerID="db4d4b6bc7082aed02445f8c95097451a32326293063d0e9ba40135d3b36ecde" Sep 10 13:37:58.845667 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:37:58.845633 2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-3ca6fbf6-5b467d68c4-g7b2q"] Sep 10 13:37:58.846135 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:37:58.845831 2557 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-mt-3ca6fbf6-5b467d68c4-g7b2q" podUID="8778a756-cdb8-4a9c-be8c-ef1a8ad373bc" containerName="payload-pre-processing" containerID="cri-o://b0afe0273015a68e9be39758483965bcb971e767382a3d58f76a4f46a0572661" gracePeriod=30 Sep 10 13:37:59.096559 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:37:59.096506 2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-3ca6fbf6-5b467d68c4-g7b2q" Sep 10 13:37:59.181914 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:37:59.181878 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-bpdxg\" (UniqueName: \"kubernetes.io/projected/8778a756-cdb8-4a9c-be8c-ef1a8ad373bc-kube-api-access-bpdxg\") pod \"8778a756-cdb8-4a9c-be8c-ef1a8ad373bc\" (UID: \"8778a756-cdb8-4a9c-be8c-ef1a8ad373bc\") " Sep 10 13:37:59.182241 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:37:59.182215 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8778a756-cdb8-4a9c-be8c-ef1a8ad373bc-plugins-config-volume\") pod \"8778a756-cdb8-4a9c-be8c-ef1a8ad373bc\" (UID: \"8778a756-cdb8-4a9c-be8c-ef1a8ad373bc\") " Sep 10 13:37:59.182847 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:37:59.182820 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8778a756-cdb8-4a9c-be8c-ef1a8ad373bc-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "8778a756-cdb8-4a9c-be8c-ef1a8ad373bc" (UID: "8778a756-cdb8-4a9c-be8c-ef1a8ad373bc"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 10 13:37:59.184103 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:37:59.184075 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8778a756-cdb8-4a9c-be8c-ef1a8ad373bc-kube-api-access-bpdxg" (OuterVolumeSpecName: "kube-api-access-bpdxg") pod "8778a756-cdb8-4a9c-be8c-ef1a8ad373bc" (UID: "8778a756-cdb8-4a9c-be8c-ef1a8ad373bc"). InnerVolumeSpecName "kube-api-access-bpdxg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 10 13:37:59.283385 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:37:59.283356 2557 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bpdxg\" (UniqueName: \"kubernetes.io/projected/8778a756-cdb8-4a9c-be8c-ef1a8ad373bc-kube-api-access-bpdxg\") on node \"ip-10-0-139-174.ec2.internal\" DevicePath \"\"" Sep 10 13:37:59.283513 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:37:59.283399 2557 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8778a756-cdb8-4a9c-be8c-ef1a8ad373bc-plugins-config-volume\") on node \"ip-10-0-139-174.ec2.internal\" DevicePath \"\"" Sep 10 13:37:59.476499 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:37:59.476466 2557 generic.go:356] "Generic (PLEG): container finished" podID="8778a756-cdb8-4a9c-be8c-ef1a8ad373bc" containerID="b0afe0273015a68e9be39758483965bcb971e767382a3d58f76a4f46a0572661" exitCode=0 Sep 10 13:37:59.476666 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:37:59.476512 2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-3ca6fbf6-5b467d68c4-g7b2q" Sep 10 13:37:59.476666 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:37:59.476559 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-3ca6fbf6-5b467d68c4-g7b2q" event={"ID":"8778a756-cdb8-4a9c-be8c-ef1a8ad373bc","Type":"ContainerDied","Data":"b0afe0273015a68e9be39758483965bcb971e767382a3d58f76a4f46a0572661"} Sep 10 13:37:59.476666 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:37:59.476612 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-3ca6fbf6-5b467d68c4-g7b2q" event={"ID":"8778a756-cdb8-4a9c-be8c-ef1a8ad373bc","Type":"ContainerDied","Data":"68c1b22418064056030f24ecd03ce4739e130a59f673c9a4305d16e177ff10f0"} Sep 10 13:37:59.476666 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:37:59.476638 2557 scope.go:117] "RemoveContainer" containerID="b0afe0273015a68e9be39758483965bcb971e767382a3d58f76a4f46a0572661" Sep 10 13:37:59.485971 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:37:59.485956 2557 scope.go:117] "RemoveContainer" containerID="b0afe0273015a68e9be39758483965bcb971e767382a3d58f76a4f46a0572661" Sep 10 13:37:59.486228 ip-10-0-139-174 kubenswrapper[2557]: E0910 13:37:59.486209 2557 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b0afe0273015a68e9be39758483965bcb971e767382a3d58f76a4f46a0572661\": container with ID starting with b0afe0273015a68e9be39758483965bcb971e767382a3d58f76a4f46a0572661 not found: ID does not exist" containerID="b0afe0273015a68e9be39758483965bcb971e767382a3d58f76a4f46a0572661" Sep 10 13:37:59.486294 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:37:59.486234 2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b0afe0273015a68e9be39758483965bcb971e767382a3d58f76a4f46a0572661"} err="failed to get container status \"b0afe0273015a68e9be39758483965bcb971e767382a3d58f76a4f46a0572661\": rpc error: code = NotFound desc = could not find container \"b0afe0273015a68e9be39758483965bcb971e767382a3d58f76a4f46a0572661\": container with ID starting with b0afe0273015a68e9be39758483965bcb971e767382a3d58f76a4f46a0572661 not found: ID does not exist" Sep 10 13:37:59.502987 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:37:59.502959 2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-3ca6fbf6-5b467d68c4-g7b2q"] Sep 10 13:37:59.515774 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:37:59.515754 2557 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-3ca6fbf6-5b467d68c4-g7b2q"] Sep 10 13:38:00.219532 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:38:00.219507 2557 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8778a756-cdb8-4a9c-be8c-ef1a8ad373bc" path="/var/lib/kubelet/pods/8778a756-cdb8-4a9c-be8c-ef1a8ad373bc/volumes" Sep 10 13:38:20.358613 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:38:20.358542 2557 scope.go:117] "RemoveContainer" containerID="543e4e1d4708b4cb66ea7b966ca706ee422276f11427d3ab638fbd36c6879cbf" Sep 10 13:38:20.369394 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:38:20.369375 2557 scope.go:117] "RemoveContainer" containerID="aa9cee7d80774c3c2aaf84cc23412eb7d5678043d6d223ca65c1f95dc117a78b" Sep 10 13:39:20.397656 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:39:20.397625 2557 scope.go:117] "RemoveContainer" containerID="3c415ef089657725f6990a29870fcb11203788ca88fa07413cfc4ecdfc339cee" Sep 10 13:40:07.285252 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:40:07.285147 2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-a-559bdf-openshift-default-665458d45d-7lc67"] Sep 10 13:40:07.285886 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:40:07.285858 2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8778a756-cdb8-4a9c-be8c-ef1a8ad373bc" containerName="payload-pre-processing" Sep 10 13:40:07.285886 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:40:07.285879 2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="8778a756-cdb8-4a9c-be8c-ef1a8ad373bc" containerName="payload-pre-processing" Sep 10 13:40:07.286010 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:40:07.285977 2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="8778a756-cdb8-4a9c-be8c-ef1a8ad373bc" containerName="payload-pre-processing" Sep 10 13:40:07.289793 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:40:07.289754 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-559bdf-openshift-default-665458d45d-7lc67" Sep 10 13:40:07.292865 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:40:07.292842 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-a-559bdf-openshift-default-dockercfg-zxtb6\"" Sep 10 13:40:07.319828 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:40:07.319804 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-559bdf-openshift-default-665458d45d-7lc67"] Sep 10 13:40:07.408982 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:40:07.408943 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cbf905c7-d254-490f-820e-83e29ff8b86d-credential-socket\") pod \"e2e-api-a-559bdf-openshift-default-665458d45d-7lc67\" (UID: \"cbf905c7-d254-490f-820e-83e29ff8b86d\") " pod="openshift-ingress/e2e-api-a-559bdf-openshift-default-665458d45d-7lc67" Sep 10 13:40:07.408982 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:40:07.408979 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cbf905c7-d254-490f-820e-83e29ff8b86d-istio-token\") pod \"e2e-api-a-559bdf-openshift-default-665458d45d-7lc67\" (UID: \"cbf905c7-d254-490f-820e-83e29ff8b86d\") " pod="openshift-ingress/e2e-api-a-559bdf-openshift-default-665458d45d-7lc67" Sep 10 13:40:07.409241 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:40:07.409104 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cbf905c7-d254-490f-820e-83e29ff8b86d-istio-envoy\") pod \"e2e-api-a-559bdf-openshift-default-665458d45d-7lc67\" (UID: \"cbf905c7-d254-490f-820e-83e29ff8b86d\") " pod="openshift-ingress/e2e-api-a-559bdf-openshift-default-665458d45d-7lc67" Sep 10 13:40:07.409241 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:40:07.409142 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cbf905c7-d254-490f-820e-83e29ff8b86d-istiod-ca-cert\") pod \"e2e-api-a-559bdf-openshift-default-665458d45d-7lc67\" (UID: \"cbf905c7-d254-490f-820e-83e29ff8b86d\") " pod="openshift-ingress/e2e-api-a-559bdf-openshift-default-665458d45d-7lc67" Sep 10 13:40:07.409241 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:40:07.409176 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cbf905c7-d254-490f-820e-83e29ff8b86d-istio-podinfo\") pod \"e2e-api-a-559bdf-openshift-default-665458d45d-7lc67\" (UID: \"cbf905c7-d254-490f-820e-83e29ff8b86d\") " pod="openshift-ingress/e2e-api-a-559bdf-openshift-default-665458d45d-7lc67" Sep 10 13:40:07.409241 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:40:07.409204 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qc8d4\" (UniqueName: \"kubernetes.io/projected/cbf905c7-d254-490f-820e-83e29ff8b86d-kube-api-access-qc8d4\") pod \"e2e-api-a-559bdf-openshift-default-665458d45d-7lc67\" (UID: \"cbf905c7-d254-490f-820e-83e29ff8b86d\") " pod="openshift-ingress/e2e-api-a-559bdf-openshift-default-665458d45d-7lc67" Sep 10 13:40:07.409241 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:40:07.409235 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cbf905c7-d254-490f-820e-83e29ff8b86d-workload-socket\") pod \"e2e-api-a-559bdf-openshift-default-665458d45d-7lc67\" (UID: \"cbf905c7-d254-490f-820e-83e29ff8b86d\") " pod="openshift-ingress/e2e-api-a-559bdf-openshift-default-665458d45d-7lc67" Sep 10 13:40:07.409511 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:40:07.409316 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cbf905c7-d254-490f-820e-83e29ff8b86d-workload-certs\") pod \"e2e-api-a-559bdf-openshift-default-665458d45d-7lc67\" (UID: \"cbf905c7-d254-490f-820e-83e29ff8b86d\") " pod="openshift-ingress/e2e-api-a-559bdf-openshift-default-665458d45d-7lc67" Sep 10 13:40:07.409511 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:40:07.409341 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cbf905c7-d254-490f-820e-83e29ff8b86d-istio-data\") pod \"e2e-api-a-559bdf-openshift-default-665458d45d-7lc67\" (UID: \"cbf905c7-d254-490f-820e-83e29ff8b86d\") " pod="openshift-ingress/e2e-api-a-559bdf-openshift-default-665458d45d-7lc67" Sep 10 13:40:07.510136 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:40:07.510108 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cbf905c7-d254-490f-820e-83e29ff8b86d-workload-socket\") pod \"e2e-api-a-559bdf-openshift-default-665458d45d-7lc67\" (UID: \"cbf905c7-d254-490f-820e-83e29ff8b86d\") " pod="openshift-ingress/e2e-api-a-559bdf-openshift-default-665458d45d-7lc67" Sep 10 13:40:07.510296 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:40:07.510156 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cbf905c7-d254-490f-820e-83e29ff8b86d-workload-certs\") pod \"e2e-api-a-559bdf-openshift-default-665458d45d-7lc67\" (UID: \"cbf905c7-d254-490f-820e-83e29ff8b86d\") " pod="openshift-ingress/e2e-api-a-559bdf-openshift-default-665458d45d-7lc67" Sep 10 13:40:07.510296 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:40:07.510174 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cbf905c7-d254-490f-820e-83e29ff8b86d-istio-data\") pod \"e2e-api-a-559bdf-openshift-default-665458d45d-7lc67\" (UID: \"cbf905c7-d254-490f-820e-83e29ff8b86d\") " pod="openshift-ingress/e2e-api-a-559bdf-openshift-default-665458d45d-7lc67" Sep 10 13:40:07.510296 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:40:07.510205 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cbf905c7-d254-490f-820e-83e29ff8b86d-credential-socket\") pod \"e2e-api-a-559bdf-openshift-default-665458d45d-7lc67\" (UID: \"cbf905c7-d254-490f-820e-83e29ff8b86d\") " pod="openshift-ingress/e2e-api-a-559bdf-openshift-default-665458d45d-7lc67" Sep 10 13:40:07.510296 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:40:07.510227 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cbf905c7-d254-490f-820e-83e29ff8b86d-istio-token\") pod \"e2e-api-a-559bdf-openshift-default-665458d45d-7lc67\" (UID: \"cbf905c7-d254-490f-820e-83e29ff8b86d\") " pod="openshift-ingress/e2e-api-a-559bdf-openshift-default-665458d45d-7lc67" Sep 10 13:40:07.510296 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:40:07.510281 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cbf905c7-d254-490f-820e-83e29ff8b86d-istio-envoy\") pod \"e2e-api-a-559bdf-openshift-default-665458d45d-7lc67\" (UID: \"cbf905c7-d254-490f-820e-83e29ff8b86d\") " pod="openshift-ingress/e2e-api-a-559bdf-openshift-default-665458d45d-7lc67" Sep 10 13:40:07.510569 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:40:07.510309 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cbf905c7-d254-490f-820e-83e29ff8b86d-istiod-ca-cert\") pod \"e2e-api-a-559bdf-openshift-default-665458d45d-7lc67\" (UID: \"cbf905c7-d254-490f-820e-83e29ff8b86d\") " pod="openshift-ingress/e2e-api-a-559bdf-openshift-default-665458d45d-7lc67" Sep 10 13:40:07.510569 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:40:07.510338 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cbf905c7-d254-490f-820e-83e29ff8b86d-istio-podinfo\") pod \"e2e-api-a-559bdf-openshift-default-665458d45d-7lc67\" (UID: \"cbf905c7-d254-490f-820e-83e29ff8b86d\") " pod="openshift-ingress/e2e-api-a-559bdf-openshift-default-665458d45d-7lc67" Sep 10 13:40:07.510569 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:40:07.510356 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qc8d4\" (UniqueName: \"kubernetes.io/projected/cbf905c7-d254-490f-820e-83e29ff8b86d-kube-api-access-qc8d4\") pod \"e2e-api-a-559bdf-openshift-default-665458d45d-7lc67\" (UID: \"cbf905c7-d254-490f-820e-83e29ff8b86d\") " pod="openshift-ingress/e2e-api-a-559bdf-openshift-default-665458d45d-7lc67" Sep 10 13:40:07.510569 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:40:07.510546 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cbf905c7-d254-490f-820e-83e29ff8b86d-workload-socket\") pod \"e2e-api-a-559bdf-openshift-default-665458d45d-7lc67\" (UID: \"cbf905c7-d254-490f-820e-83e29ff8b86d\") " pod="openshift-ingress/e2e-api-a-559bdf-openshift-default-665458d45d-7lc67" Sep 10 13:40:07.510742 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:40:07.510723 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cbf905c7-d254-490f-820e-83e29ff8b86d-istio-data\") pod \"e2e-api-a-559bdf-openshift-default-665458d45d-7lc67\" (UID: \"cbf905c7-d254-490f-820e-83e29ff8b86d\") " pod="openshift-ingress/e2e-api-a-559bdf-openshift-default-665458d45d-7lc67" Sep 10 13:40:07.510816 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:40:07.510795 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cbf905c7-d254-490f-820e-83e29ff8b86d-workload-certs\") pod \"e2e-api-a-559bdf-openshift-default-665458d45d-7lc67\" (UID: \"cbf905c7-d254-490f-820e-83e29ff8b86d\") " pod="openshift-ingress/e2e-api-a-559bdf-openshift-default-665458d45d-7lc67" Sep 10 13:40:07.510866 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:40:07.510840 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cbf905c7-d254-490f-820e-83e29ff8b86d-credential-socket\") pod \"e2e-api-a-559bdf-openshift-default-665458d45d-7lc67\" (UID: \"cbf905c7-d254-490f-820e-83e29ff8b86d\") " pod="openshift-ingress/e2e-api-a-559bdf-openshift-default-665458d45d-7lc67" Sep 10 13:40:07.511066 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:40:07.511042 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cbf905c7-d254-490f-820e-83e29ff8b86d-istiod-ca-cert\") pod \"e2e-api-a-559bdf-openshift-default-665458d45d-7lc67\" (UID: \"cbf905c7-d254-490f-820e-83e29ff8b86d\") " pod="openshift-ingress/e2e-api-a-559bdf-openshift-default-665458d45d-7lc67" Sep 10 13:40:07.512704 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:40:07.512679 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cbf905c7-d254-490f-820e-83e29ff8b86d-istio-envoy\") pod \"e2e-api-a-559bdf-openshift-default-665458d45d-7lc67\" (UID: \"cbf905c7-d254-490f-820e-83e29ff8b86d\") " pod="openshift-ingress/e2e-api-a-559bdf-openshift-default-665458d45d-7lc67" Sep 10 13:40:07.512962 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:40:07.512946 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cbf905c7-d254-490f-820e-83e29ff8b86d-istio-podinfo\") pod \"e2e-api-a-559bdf-openshift-default-665458d45d-7lc67\" (UID: \"cbf905c7-d254-490f-820e-83e29ff8b86d\") " pod="openshift-ingress/e2e-api-a-559bdf-openshift-default-665458d45d-7lc67" Sep 10 13:40:07.518100 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:40:07.518069 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qc8d4\" (UniqueName: \"kubernetes.io/projected/cbf905c7-d254-490f-820e-83e29ff8b86d-kube-api-access-qc8d4\") pod \"e2e-api-a-559bdf-openshift-default-665458d45d-7lc67\" (UID: \"cbf905c7-d254-490f-820e-83e29ff8b86d\") " pod="openshift-ingress/e2e-api-a-559bdf-openshift-default-665458d45d-7lc67" Sep 10 13:40:07.518550 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:40:07.518526 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cbf905c7-d254-490f-820e-83e29ff8b86d-istio-token\") pod \"e2e-api-a-559bdf-openshift-default-665458d45d-7lc67\" (UID: \"cbf905c7-d254-490f-820e-83e29ff8b86d\") " pod="openshift-ingress/e2e-api-a-559bdf-openshift-default-665458d45d-7lc67" Sep 10 13:40:07.600047 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:40:07.599966 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-559bdf-openshift-default-665458d45d-7lc67" Sep 10 13:40:07.899670 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:40:07.899644 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-559bdf-openshift-default-665458d45d-7lc67"] Sep 10 13:40:07.899934 ip-10-0-139-174 kubenswrapper[2557]: W0910 13:40:07.899911 2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcbf905c7_d254_490f_820e_83e29ff8b86d.slice/crio-d2a79a7d4c2e11a9ad68cdbcb0b44fafd249f03fe3c8823291aff91ef98ad9b9 WatchSource:0}: Error finding container d2a79a7d4c2e11a9ad68cdbcb0b44fafd249f03fe3c8823291aff91ef98ad9b9: Status 404 returned error can't find the container with id d2a79a7d4c2e11a9ad68cdbcb0b44fafd249f03fe3c8823291aff91ef98ad9b9 Sep 10 13:40:07.902283 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:40:07.902250 2557 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892152Ki","pods":"250"} Sep 10 13:40:07.902385 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:40:07.902328 2557 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892152Ki","pods":"250"} Sep 10 13:40:07.902385 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:40:07.902375 2557 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892152Ki","pods":"250"} Sep 10 13:40:08.235494 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:40:08.235454 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-559bdf-openshift-default-665458d45d-7lc67" event={"ID":"cbf905c7-d254-490f-820e-83e29ff8b86d","Type":"ContainerStarted","Data":"e2352722942044761ba03fe59fba912f2ac64597061392d075b001ac3073a016"} Sep 10 13:40:08.235658 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:40:08.235518 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-559bdf-openshift-default-665458d45d-7lc67" event={"ID":"cbf905c7-d254-490f-820e-83e29ff8b86d","Type":"ContainerStarted","Data":"d2a79a7d4c2e11a9ad68cdbcb0b44fafd249f03fe3c8823291aff91ef98ad9b9"} Sep 10 13:40:08.261861 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:40:08.261800 2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-api-a-559bdf-openshift-default-665458d45d-7lc67" podStartSLOduration=1.26178301 podStartE2EDuration="1.26178301s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 13:40:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 13:40:08.25915903 +0000 UTC m=+1368.566153445" watchObservedRunningTime="2026-09-10 13:40:08.26178301 +0000 UTC m=+1368.568777425" Sep 10 13:40:08.600359 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:40:08.600277 2557 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-559bdf-openshift-default-665458d45d-7lc67" Sep 10 13:40:08.606085 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:40:08.606060 2557 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-559bdf-openshift-default-665458d45d-7lc67" Sep 10 13:40:09.242312 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:40:09.242277 2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-559bdf-openshift-default-665458d45d-7lc67" Sep 10 13:40:09.243184 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:40:09.243165 2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-559bdf-openshift-default-665458d45d-7lc67" Sep 10 13:40:25.283609 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:40:25.283563 2557 patch_prober.go:28] interesting pod/e2e-api-a-559bdf-openshift-default-665458d45d-7lc67 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.72:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Sep 10 13:40:25.284083 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:40:25.283640 2557 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-a-559bdf-openshift-default-665458d45d-7lc67" podUID="cbf905c7-d254-490f-820e-83e29ff8b86d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.72:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Sep 10 13:40:28.356262 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:40:28.356231 2557 generic.go:356] "Generic (PLEG): container finished" podID="cbf905c7-d254-490f-820e-83e29ff8b86d" containerID="e2352722942044761ba03fe59fba912f2ac64597061392d075b001ac3073a016" exitCode=0 Sep 10 13:40:28.356715 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:40:28.356296 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-559bdf-openshift-default-665458d45d-7lc67" event={"ID":"cbf905c7-d254-490f-820e-83e29ff8b86d","Type":"ContainerDied","Data":"e2352722942044761ba03fe59fba912f2ac64597061392d075b001ac3073a016"} Sep 10 13:40:28.356797 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:40:28.356770 2557 scope.go:117] "RemoveContainer" containerID="e2352722942044761ba03fe59fba912f2ac64597061392d075b001ac3073a016" Sep 10 13:40:28.357906 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:40:28.357881 2557 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892152Ki","pods":"250"} Sep 10 13:40:28.357956 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:40:28.357944 2557 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892152Ki","pods":"250"} Sep 10 13:40:28.357991 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:40:28.357975 2557 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892152Ki","pods":"250"} Sep 10 13:40:29.364927 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:40:29.364894 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-559bdf-openshift-default-665458d45d-7lc67" event={"ID":"cbf905c7-d254-490f-820e-83e29ff8b86d","Type":"ContainerStarted","Data":"d7417d4f3718eea48c6971cf35d3fc1ee371f77f0a121d1d63150986913e6211"} Sep 10 13:40:29.600183 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:40:29.600144 2557 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-559bdf-openshift-default-665458d45d-7lc67" Sep 10 13:40:29.600384 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:40:29.600339 2557 patch_prober.go:28] interesting pod/e2e-api-a-559bdf-openshift-default-665458d45d-7lc67 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.72:15021/healthz/ready\": dial tcp 10.134.0.72:15021: connect: connection refused" start-of-body= Sep 10 13:40:29.600493 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:40:29.600393 2557 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-559bdf-openshift-default-665458d45d-7lc67" podUID="cbf905c7-d254-490f-820e-83e29ff8b86d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.72:15021/healthz/ready\": dial tcp 10.134.0.72:15021: connect: connection refused" Sep 10 13:40:30.600903 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:40:30.600869 2557 patch_prober.go:28] interesting pod/e2e-api-a-559bdf-openshift-default-665458d45d-7lc67 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.72:15021/healthz/ready\": dial tcp 10.134.0.72:15021: connect: connection refused" start-of-body= Sep 10 13:40:30.601291 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:40:30.600942 2557 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-559bdf-openshift-default-665458d45d-7lc67" podUID="cbf905c7-d254-490f-820e-83e29ff8b86d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.72:15021/healthz/ready\": dial tcp 10.134.0.72:15021: connect: connection refused" Sep 10 13:40:31.600572 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:40:31.600533 2557 patch_prober.go:28] interesting pod/e2e-api-a-559bdf-openshift-default-665458d45d-7lc67 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.72:15021/healthz/ready\": dial tcp 10.134.0.72:15021: connect: connection refused" start-of-body= Sep 10 13:40:31.600758 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:40:31.600590 2557 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-559bdf-openshift-default-665458d45d-7lc67" podUID="cbf905c7-d254-490f-820e-83e29ff8b86d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.72:15021/healthz/ready\": dial tcp 10.134.0.72:15021: connect: connection refused" Sep 10 13:40:32.601192 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:40:32.601159 2557 patch_prober.go:28] interesting pod/e2e-api-a-559bdf-openshift-default-665458d45d-7lc67 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.72:15021/healthz/ready\": dial tcp 10.134.0.72:15021: connect: connection refused" start-of-body= Sep 10 13:40:32.601597 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:40:32.601214 2557 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-559bdf-openshift-default-665458d45d-7lc67" podUID="cbf905c7-d254-490f-820e-83e29ff8b86d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.72:15021/healthz/ready\": dial tcp 10.134.0.72:15021: connect: connection refused" Sep 10 13:40:33.600852 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:40:33.600811 2557 patch_prober.go:28] interesting pod/e2e-api-a-559bdf-openshift-default-665458d45d-7lc67 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.72:15021/healthz/ready\": dial tcp 10.134.0.72:15021: connect: connection refused" start-of-body= Sep 10 13:40:33.601027 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:40:33.600869 2557 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-559bdf-openshift-default-665458d45d-7lc67" podUID="cbf905c7-d254-490f-820e-83e29ff8b86d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.72:15021/healthz/ready\": dial tcp 10.134.0.72:15021: connect: connection refused" Sep 10 13:40:34.601016 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:40:34.600981 2557 patch_prober.go:28] interesting pod/e2e-api-a-559bdf-openshift-default-665458d45d-7lc67 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.72:15021/healthz/ready\": dial tcp 10.134.0.72:15021: connect: connection refused" start-of-body= Sep 10 13:40:34.601469 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:40:34.601039 2557 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-559bdf-openshift-default-665458d45d-7lc67" podUID="cbf905c7-d254-490f-820e-83e29ff8b86d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.72:15021/healthz/ready\": dial tcp 10.134.0.72:15021: connect: connection refused" Sep 10 13:40:35.604433 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:40:35.604394 2557 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-559bdf-openshift-default-665458d45d-7lc67" Sep 10 13:40:35.604919 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:40:35.604906 2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-559bdf-openshift-default-665458d45d-7lc67" Sep 10 13:40:35.605580 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:40:35.605560 2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-559bdf-openshift-default-665458d45d-7lc67" Sep 10 13:40:36.586791 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:40:36.586760 2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-b-fa4d53-openshift-default-67c485f77b-l5b79"] Sep 10 13:40:36.589852 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:40:36.589830 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-fa4d53-openshift-default-67c485f77b-l5b79" Sep 10 13:40:36.592183 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:40:36.592160 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-b-fa4d53-openshift-default-dockercfg-8wtgd\"" Sep 10 13:40:36.604689 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:40:36.604666 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-fa4d53-openshift-default-67c485f77b-l5b79"] Sep 10 13:40:36.622876 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:40:36.622849 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e39de022-9276-43ee-a39f-875f31e695a0-credential-socket\") pod \"e2e-api-b-fa4d53-openshift-default-67c485f77b-l5b79\" (UID: \"e39de022-9276-43ee-a39f-875f31e695a0\") " pod="openshift-ingress/e2e-api-b-fa4d53-openshift-default-67c485f77b-l5b79" Sep 10 13:40:36.622983 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:40:36.622886 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e39de022-9276-43ee-a39f-875f31e695a0-istio-token\") pod \"e2e-api-b-fa4d53-openshift-default-67c485f77b-l5b79\" (UID: \"e39de022-9276-43ee-a39f-875f31e695a0\") " pod="openshift-ingress/e2e-api-b-fa4d53-openshift-default-67c485f77b-l5b79" Sep 10 13:40:36.622983 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:40:36.622917 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e39de022-9276-43ee-a39f-875f31e695a0-istio-podinfo\") pod \"e2e-api-b-fa4d53-openshift-default-67c485f77b-l5b79\" (UID: \"e39de022-9276-43ee-a39f-875f31e695a0\") " pod="openshift-ingress/e2e-api-b-fa4d53-openshift-default-67c485f77b-l5b79" Sep 10 13:40:36.623105 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:40:36.622994 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e39de022-9276-43ee-a39f-875f31e695a0-workload-certs\") pod \"e2e-api-b-fa4d53-openshift-default-67c485f77b-l5b79\" (UID: \"e39de022-9276-43ee-a39f-875f31e695a0\") " pod="openshift-ingress/e2e-api-b-fa4d53-openshift-default-67c485f77b-l5b79" Sep 10 13:40:36.623105 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:40:36.623023 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6xpzm\" (UniqueName: \"kubernetes.io/projected/e39de022-9276-43ee-a39f-875f31e695a0-kube-api-access-6xpzm\") pod \"e2e-api-b-fa4d53-openshift-default-67c485f77b-l5b79\" (UID: \"e39de022-9276-43ee-a39f-875f31e695a0\") " pod="openshift-ingress/e2e-api-b-fa4d53-openshift-default-67c485f77b-l5b79" Sep 10 13:40:36.623105 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:40:36.623046 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e39de022-9276-43ee-a39f-875f31e695a0-istiod-ca-cert\") pod \"e2e-api-b-fa4d53-openshift-default-67c485f77b-l5b79\" (UID: \"e39de022-9276-43ee-a39f-875f31e695a0\") " pod="openshift-ingress/e2e-api-b-fa4d53-openshift-default-67c485f77b-l5b79" Sep 10 13:40:36.623105 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:40:36.623092 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e39de022-9276-43ee-a39f-875f31e695a0-istio-data\") pod \"e2e-api-b-fa4d53-openshift-default-67c485f77b-l5b79\" (UID: \"e39de022-9276-43ee-a39f-875f31e695a0\") " pod="openshift-ingress/e2e-api-b-fa4d53-openshift-default-67c485f77b-l5b79" Sep 10 13:40:36.623239 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:40:36.623159 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e39de022-9276-43ee-a39f-875f31e695a0-istio-envoy\") pod \"e2e-api-b-fa4d53-openshift-default-67c485f77b-l5b79\" (UID: \"e39de022-9276-43ee-a39f-875f31e695a0\") " pod="openshift-ingress/e2e-api-b-fa4d53-openshift-default-67c485f77b-l5b79" Sep 10 13:40:36.623239 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:40:36.623196 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e39de022-9276-43ee-a39f-875f31e695a0-workload-socket\") pod \"e2e-api-b-fa4d53-openshift-default-67c485f77b-l5b79\" (UID: \"e39de022-9276-43ee-a39f-875f31e695a0\") " pod="openshift-ingress/e2e-api-b-fa4d53-openshift-default-67c485f77b-l5b79" Sep 10 13:40:36.724727 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:40:36.724685 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e39de022-9276-43ee-a39f-875f31e695a0-istio-envoy\") pod \"e2e-api-b-fa4d53-openshift-default-67c485f77b-l5b79\" (UID: \"e39de022-9276-43ee-a39f-875f31e695a0\") " pod="openshift-ingress/e2e-api-b-fa4d53-openshift-default-67c485f77b-l5b79" Sep 10 13:40:36.724933 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:40:36.724753 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e39de022-9276-43ee-a39f-875f31e695a0-workload-socket\") pod \"e2e-api-b-fa4d53-openshift-default-67c485f77b-l5b79\" (UID: \"e39de022-9276-43ee-a39f-875f31e695a0\") " pod="openshift-ingress/e2e-api-b-fa4d53-openshift-default-67c485f77b-l5b79" Sep 10 13:40:36.724933 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:40:36.724868 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e39de022-9276-43ee-a39f-875f31e695a0-credential-socket\") pod \"e2e-api-b-fa4d53-openshift-default-67c485f77b-l5b79\" (UID: \"e39de022-9276-43ee-a39f-875f31e695a0\") " pod="openshift-ingress/e2e-api-b-fa4d53-openshift-default-67c485f77b-l5b79" Sep 10 13:40:36.724933 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:40:36.724892 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e39de022-9276-43ee-a39f-875f31e695a0-istio-token\") pod \"e2e-api-b-fa4d53-openshift-default-67c485f77b-l5b79\" (UID: \"e39de022-9276-43ee-a39f-875f31e695a0\") " pod="openshift-ingress/e2e-api-b-fa4d53-openshift-default-67c485f77b-l5b79" Sep 10 13:40:36.724933 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:40:36.724924 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e39de022-9276-43ee-a39f-875f31e695a0-istio-podinfo\") pod \"e2e-api-b-fa4d53-openshift-default-67c485f77b-l5b79\" (UID: \"e39de022-9276-43ee-a39f-875f31e695a0\") " pod="openshift-ingress/e2e-api-b-fa4d53-openshift-default-67c485f77b-l5b79" Sep 10 13:40:36.725155 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:40:36.724967 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e39de022-9276-43ee-a39f-875f31e695a0-workload-certs\") pod \"e2e-api-b-fa4d53-openshift-default-67c485f77b-l5b79\" (UID: \"e39de022-9276-43ee-a39f-875f31e695a0\") " pod="openshift-ingress/e2e-api-b-fa4d53-openshift-default-67c485f77b-l5b79" Sep 10 13:40:36.725155 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:40:36.724991 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6xpzm\" (UniqueName: \"kubernetes.io/projected/e39de022-9276-43ee-a39f-875f31e695a0-kube-api-access-6xpzm\") pod \"e2e-api-b-fa4d53-openshift-default-67c485f77b-l5b79\" (UID: \"e39de022-9276-43ee-a39f-875f31e695a0\") " pod="openshift-ingress/e2e-api-b-fa4d53-openshift-default-67c485f77b-l5b79" Sep 10 13:40:36.725155 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:40:36.725021 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e39de022-9276-43ee-a39f-875f31e695a0-istiod-ca-cert\") pod \"e2e-api-b-fa4d53-openshift-default-67c485f77b-l5b79\" (UID: \"e39de022-9276-43ee-a39f-875f31e695a0\") " pod="openshift-ingress/e2e-api-b-fa4d53-openshift-default-67c485f77b-l5b79" Sep 10 13:40:36.725155 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:40:36.725052 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e39de022-9276-43ee-a39f-875f31e695a0-istio-data\") pod \"e2e-api-b-fa4d53-openshift-default-67c485f77b-l5b79\" (UID: \"e39de022-9276-43ee-a39f-875f31e695a0\") " pod="openshift-ingress/e2e-api-b-fa4d53-openshift-default-67c485f77b-l5b79" Sep 10 13:40:36.725555 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:40:36.725527 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e39de022-9276-43ee-a39f-875f31e695a0-istio-data\") pod \"e2e-api-b-fa4d53-openshift-default-67c485f77b-l5b79\" (UID: \"e39de022-9276-43ee-a39f-875f31e695a0\") " pod="openshift-ingress/e2e-api-b-fa4d53-openshift-default-67c485f77b-l5b79" Sep 10 13:40:36.726735 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:40:36.726691 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e39de022-9276-43ee-a39f-875f31e695a0-workload-socket\") pod \"e2e-api-b-fa4d53-openshift-default-67c485f77b-l5b79\" (UID: \"e39de022-9276-43ee-a39f-875f31e695a0\") " pod="openshift-ingress/e2e-api-b-fa4d53-openshift-default-67c485f77b-l5b79" Sep 10 13:40:36.727118 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:40:36.727097 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e39de022-9276-43ee-a39f-875f31e695a0-credential-socket\") pod \"e2e-api-b-fa4d53-openshift-default-67c485f77b-l5b79\" (UID: \"e39de022-9276-43ee-a39f-875f31e695a0\") " pod="openshift-ingress/e2e-api-b-fa4d53-openshift-default-67c485f77b-l5b79" Sep 10 13:40:36.727291 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:40:36.727236 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e39de022-9276-43ee-a39f-875f31e695a0-istiod-ca-cert\") pod \"e2e-api-b-fa4d53-openshift-default-67c485f77b-l5b79\" (UID: \"e39de022-9276-43ee-a39f-875f31e695a0\") " pod="openshift-ingress/e2e-api-b-fa4d53-openshift-default-67c485f77b-l5b79" Sep 10 13:40:36.728070 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:40:36.728049 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e39de022-9276-43ee-a39f-875f31e695a0-workload-certs\") pod \"e2e-api-b-fa4d53-openshift-default-67c485f77b-l5b79\" (UID: \"e39de022-9276-43ee-a39f-875f31e695a0\") " pod="openshift-ingress/e2e-api-b-fa4d53-openshift-default-67c485f77b-l5b79" Sep 10 13:40:36.729392 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:40:36.728697 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e39de022-9276-43ee-a39f-875f31e695a0-istio-envoy\") pod \"e2e-api-b-fa4d53-openshift-default-67c485f77b-l5b79\" (UID: \"e39de022-9276-43ee-a39f-875f31e695a0\") " pod="openshift-ingress/e2e-api-b-fa4d53-openshift-default-67c485f77b-l5b79" Sep 10 13:40:36.729392 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:40:36.728981 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e39de022-9276-43ee-a39f-875f31e695a0-istio-podinfo\") pod \"e2e-api-b-fa4d53-openshift-default-67c485f77b-l5b79\" (UID: \"e39de022-9276-43ee-a39f-875f31e695a0\") " pod="openshift-ingress/e2e-api-b-fa4d53-openshift-default-67c485f77b-l5b79" Sep 10 13:40:36.736894 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:40:36.736865 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6xpzm\" (UniqueName: \"kubernetes.io/projected/e39de022-9276-43ee-a39f-875f31e695a0-kube-api-access-6xpzm\") pod \"e2e-api-b-fa4d53-openshift-default-67c485f77b-l5b79\" (UID: \"e39de022-9276-43ee-a39f-875f31e695a0\") " pod="openshift-ingress/e2e-api-b-fa4d53-openshift-default-67c485f77b-l5b79" Sep 10 13:40:36.737029 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:40:36.737009 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e39de022-9276-43ee-a39f-875f31e695a0-istio-token\") pod \"e2e-api-b-fa4d53-openshift-default-67c485f77b-l5b79\" (UID: \"e39de022-9276-43ee-a39f-875f31e695a0\") " pod="openshift-ingress/e2e-api-b-fa4d53-openshift-default-67c485f77b-l5b79" Sep 10 13:40:36.903320 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:40:36.903249 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-fa4d53-openshift-default-67c485f77b-l5b79" Sep 10 13:40:37.219508 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:40:37.219487 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-fa4d53-openshift-default-67c485f77b-l5b79"] Sep 10 13:40:37.221490 ip-10-0-139-174 kubenswrapper[2557]: W0910 13:40:37.221462 2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode39de022_9276_43ee_a39f_875f31e695a0.slice/crio-33c094c7fedbf42054ffd4bf2f3982c3838ec6d668e21a8db9cc7cd02bbff8f9 WatchSource:0}: Error finding container 33c094c7fedbf42054ffd4bf2f3982c3838ec6d668e21a8db9cc7cd02bbff8f9: Status 404 returned error can't find the container with id 33c094c7fedbf42054ffd4bf2f3982c3838ec6d668e21a8db9cc7cd02bbff8f9 Sep 10 13:40:37.223712 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:40:37.223677 2557 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892152Ki","pods":"250"} Sep 10 13:40:37.223812 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:40:37.223741 2557 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892152Ki","pods":"250"} Sep 10 13:40:37.223812 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:40:37.223779 2557 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892152Ki","pods":"250"} Sep 10 13:40:37.414703 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:40:37.414665 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-fa4d53-openshift-default-67c485f77b-l5b79" event={"ID":"e39de022-9276-43ee-a39f-875f31e695a0","Type":"ContainerStarted","Data":"ea17cc3f405ab5366dd61861c8d5f4c92c56e5de8eda6382c26e2a5d595e91c4"} Sep 10 13:40:37.414926 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:40:37.414735 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-fa4d53-openshift-default-67c485f77b-l5b79" event={"ID":"e39de022-9276-43ee-a39f-875f31e695a0","Type":"ContainerStarted","Data":"33c094c7fedbf42054ffd4bf2f3982c3838ec6d668e21a8db9cc7cd02bbff8f9"} Sep 10 13:40:37.438523 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:40:37.438469 2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-api-b-fa4d53-openshift-default-67c485f77b-l5b79" podStartSLOduration=1.438448227 podStartE2EDuration="1.438448227s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 13:40:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 13:40:37.43514326 +0000 UTC m=+1397.742137676" watchObservedRunningTime="2026-09-10 13:40:37.438448227 +0000 UTC m=+1397.745442636" Sep 10 13:40:37.903884 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:40:37.903846 2557 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-fa4d53-openshift-default-67c485f77b-l5b79" Sep 10 13:40:38.909158 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:40:38.909131 2557 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-fa4d53-openshift-default-67c485f77b-l5b79" Sep 10 13:40:39.426762 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:40:39.426726 2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-fa4d53-openshift-default-67c485f77b-l5b79" Sep 10 13:40:39.427620 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:40:39.427597 2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-fa4d53-openshift-default-67c485f77b-l5b79" Sep 10 13:40:59.542353 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:40:59.542321 2557 generic.go:356] "Generic (PLEG): container finished" podID="e39de022-9276-43ee-a39f-875f31e695a0" containerID="ea17cc3f405ab5366dd61861c8d5f4c92c56e5de8eda6382c26e2a5d595e91c4" exitCode=0 Sep 10 13:40:59.542755 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:40:59.542395 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-fa4d53-openshift-default-67c485f77b-l5b79" event={"ID":"e39de022-9276-43ee-a39f-875f31e695a0","Type":"ContainerDied","Data":"ea17cc3f405ab5366dd61861c8d5f4c92c56e5de8eda6382c26e2a5d595e91c4"} Sep 10 13:40:59.542808 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:40:59.542780 2557 scope.go:117] "RemoveContainer" containerID="ea17cc3f405ab5366dd61861c8d5f4c92c56e5de8eda6382c26e2a5d595e91c4" Sep 10 13:40:59.543875 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:40:59.543848 2557 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892152Ki","pods":"250"} Sep 10 13:40:59.543983 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:40:59.543909 2557 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892152Ki","pods":"250"} Sep 10 13:40:59.543983 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:40:59.543939 2557 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892152Ki","pods":"250"} Sep 10 13:41:00.555894 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:41:00.555851 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-fa4d53-openshift-default-67c485f77b-l5b79" event={"ID":"e39de022-9276-43ee-a39f-875f31e695a0","Type":"ContainerStarted","Data":"3884b5f75442f5aa3d831df266642716f1c0db41c71416a2478fb6401e70741a"} Sep 10 13:41:00.904126 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:41:00.904031 2557 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-fa4d53-openshift-default-67c485f77b-l5b79" Sep 10 13:41:00.904315 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:41:00.904237 2557 patch_prober.go:28] interesting pod/e2e-api-b-fa4d53-openshift-default-67c485f77b-l5b79 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.73:15021/healthz/ready\": dial tcp 10.134.0.73:15021: connect: connection refused" start-of-body= Sep 10 13:41:00.904384 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:41:00.904309 2557 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-fa4d53-openshift-default-67c485f77b-l5b79" podUID="e39de022-9276-43ee-a39f-875f31e695a0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.73:15021/healthz/ready\": dial tcp 10.134.0.73:15021: connect: connection refused" Sep 10 13:41:01.904239 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:41:01.904206 2557 patch_prober.go:28] interesting pod/e2e-api-b-fa4d53-openshift-default-67c485f77b-l5b79 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.73:15021/healthz/ready\": dial tcp 10.134.0.73:15021: connect: connection refused" start-of-body= Sep 10 13:41:01.904632 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:41:01.904262 2557 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-fa4d53-openshift-default-67c485f77b-l5b79" podUID="e39de022-9276-43ee-a39f-875f31e695a0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.73:15021/healthz/ready\": dial tcp 10.134.0.73:15021: connect: connection refused" Sep 10 13:41:02.904229 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:41:02.904189 2557 patch_prober.go:28] interesting pod/e2e-api-b-fa4d53-openshift-default-67c485f77b-l5b79 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.73:15021/healthz/ready\": dial tcp 10.134.0.73:15021: connect: connection refused" start-of-body= Sep 10 13:41:02.904465 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:41:02.904261 2557 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-fa4d53-openshift-default-67c485f77b-l5b79" podUID="e39de022-9276-43ee-a39f-875f31e695a0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.73:15021/healthz/ready\": dial tcp 10.134.0.73:15021: connect: connection refused" Sep 10 13:41:03.904126 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:41:03.904091 2557 patch_prober.go:28] interesting pod/e2e-api-b-fa4d53-openshift-default-67c485f77b-l5b79 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.73:15021/healthz/ready\": dial tcp 10.134.0.73:15021: connect: connection refused" start-of-body= Sep 10 13:41:03.904307 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:41:03.904140 2557 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-fa4d53-openshift-default-67c485f77b-l5b79" podUID="e39de022-9276-43ee-a39f-875f31e695a0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.73:15021/healthz/ready\": dial tcp 10.134.0.73:15021: connect: connection refused" Sep 10 13:41:04.903664 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:41:04.903633 2557 patch_prober.go:28] interesting pod/e2e-api-b-fa4d53-openshift-default-67c485f77b-l5b79 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.73:15021/healthz/ready\": dial tcp 10.134.0.73:15021: connect: connection refused" start-of-body= Sep 10 13:41:04.904035 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:41:04.903692 2557 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-fa4d53-openshift-default-67c485f77b-l5b79" podUID="e39de022-9276-43ee-a39f-875f31e695a0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.73:15021/healthz/ready\": dial tcp 10.134.0.73:15021: connect: connection refused" Sep 10 13:41:06.042243 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:41:06.042208 2557 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-fa4d53-openshift-default-67c485f77b-l5b79" Sep 10 13:41:06.042757 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:41:06.042736 2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-fa4d53-openshift-default-67c485f77b-l5b79" Sep 10 13:41:06.044828 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:41:06.044794 2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-fa4d53-openshift-default-67c485f77b-l5b79" Sep 10 13:41:08.034124 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:41:08.034085 2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-559bdf-qz4mr"] Sep 10 13:41:08.037063 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:41:08.037040 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-559bdf-qz4mr" Sep 10 13:41:08.042273 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:41:08.042253 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-wtblv\"" Sep 10 13:41:08.042443 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:41:08.042402 2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Sep 10 13:41:08.061809 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:41:08.061788 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-559bdf-qz4mr"] Sep 10 13:41:08.129624 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:41:08.129592 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/fe164b0e-a98d-4992-baa2-090cde904ab1-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-a-559bdf-qz4mr\" (UID: \"fe164b0e-a98d-4992-baa2-090cde904ab1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-559bdf-qz4mr" Sep 10 13:41:08.232890 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:41:08.232843 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/fe164b0e-a98d-4992-baa2-090cde904ab1-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-a-559bdf-qz4mr\" (UID: \"fe164b0e-a98d-4992-baa2-090cde904ab1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-559bdf-qz4mr" Sep 10 13:41:08.233632 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:41:08.233606 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/fe164b0e-a98d-4992-baa2-090cde904ab1-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-a-559bdf-qz4mr\" (UID: \"fe164b0e-a98d-4992-baa2-090cde904ab1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-559bdf-qz4mr" Sep 10 13:41:08.348068 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:41:08.347990 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-559bdf-qz4mr" Sep 10 13:41:08.625062 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:41:08.625034 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-559bdf-qz4mr"] Sep 10 13:41:08.626625 ip-10-0-139-174 kubenswrapper[2557]: W0910 13:41:08.626591 2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfe164b0e_a98d_4992_baa2_090cde904ab1.slice/crio-1999d9c7ac67aab6477525947c3d9f5083b91c4cdd6dc21380992a86b706d586 WatchSource:0}: Error finding container 1999d9c7ac67aab6477525947c3d9f5083b91c4cdd6dc21380992a86b706d586: Status 404 returned error can't find the container with id 1999d9c7ac67aab6477525947c3d9f5083b91c4cdd6dc21380992a86b706d586 Sep 10 13:41:09.617354 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:41:09.617322 2557 generic.go:356] "Generic (PLEG): container finished" podID="fe164b0e-a98d-4992-baa2-090cde904ab1" containerID="c6c0cb089164246159b2402f9b049d7d596a0c3f64bd26f7aeae156dae073fbe" exitCode=0 Sep 10 13:41:09.617762 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:41:09.617411 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-559bdf-qz4mr" event={"ID":"fe164b0e-a98d-4992-baa2-090cde904ab1","Type":"ContainerDied","Data":"c6c0cb089164246159b2402f9b049d7d596a0c3f64bd26f7aeae156dae073fbe"} Sep 10 13:41:09.617762 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:41:09.617484 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-559bdf-qz4mr" event={"ID":"fe164b0e-a98d-4992-baa2-090cde904ab1","Type":"ContainerStarted","Data":"1999d9c7ac67aab6477525947c3d9f5083b91c4cdd6dc21380992a86b706d586"} Sep 10 13:41:10.748257 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:41:10.748230 2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-559bdf-qz4mr" Sep 10 13:41:10.854859 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:41:10.854827 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/fe164b0e-a98d-4992-baa2-090cde904ab1-maas-api-service-ca\") pod \"fe164b0e-a98d-4992-baa2-090cde904ab1\" (UID: \"fe164b0e-a98d-4992-baa2-090cde904ab1\") " Sep 10 13:41:10.855133 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:41:10.855112 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fe164b0e-a98d-4992-baa2-090cde904ab1-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "fe164b0e-a98d-4992-baa2-090cde904ab1" (UID: "fe164b0e-a98d-4992-baa2-090cde904ab1"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 10 13:41:10.855243 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:41:10.855230 2557 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/fe164b0e-a98d-4992-baa2-090cde904ab1-maas-api-service-ca\") on node \"ip-10-0-139-174.ec2.internal\" DevicePath \"\"" Sep 10 13:41:11.630638 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:41:11.630566 2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-559bdf-qz4mr" Sep 10 13:41:11.630638 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:41:11.630610 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-559bdf-qz4mr" event={"ID":"fe164b0e-a98d-4992-baa2-090cde904ab1","Type":"ContainerDied","Data":"1999d9c7ac67aab6477525947c3d9f5083b91c4cdd6dc21380992a86b706d586"} Sep 10 13:41:11.630842 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:41:11.630657 2557 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="1999d9c7ac67aab6477525947c3d9f5083b91c4cdd6dc21380992a86b706d586" Sep 10 13:41:20.430850 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:41:20.430822 2557 scope.go:117] "RemoveContainer" containerID="d8475533d299565f50768b063a38304850578f83f458a75ff3d4eedca28a2fbe" Sep 10 13:41:24.186817 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:41:24.186784 2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-559bdf-qz4mr"] Sep 10 13:41:24.192811 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:41:24.192789 2557 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-559bdf-qz4mr"] Sep 10 13:41:24.220282 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:41:24.220248 2557 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="fe164b0e-a98d-4992-baa2-090cde904ab1" path="/var/lib/kubelet/pods/fe164b0e-a98d-4992-baa2-090cde904ab1/volumes" Sep 10 13:41:33.235181 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:41:33.235147 2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-559bdf-openshift-default-665458d45d-7lc67"] Sep 10 13:41:33.235659 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:41:33.235368 2557 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-api-a-559bdf-openshift-default-665458d45d-7lc67" podUID="cbf905c7-d254-490f-820e-83e29ff8b86d" containerName="istio-proxy" containerID="cri-o://d7417d4f3718eea48c6971cf35d3fc1ee371f77f0a121d1d63150986913e6211" gracePeriod=30 Sep 10 13:41:33.626689 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:41:33.626610 2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-fa4d53-9hzcl"] Sep 10 13:41:33.627028 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:41:33.627016 2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fe164b0e-a98d-4992-baa2-090cde904ab1" containerName="revoke-keys" Sep 10 13:41:33.627071 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:41:33.627031 2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="fe164b0e-a98d-4992-baa2-090cde904ab1" containerName="revoke-keys" Sep 10 13:41:33.627122 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:41:33.627113 2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="fe164b0e-a98d-4992-baa2-090cde904ab1" containerName="revoke-keys" Sep 10 13:41:33.630175 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:41:33.630156 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-fa4d53-9hzcl" Sep 10 13:41:33.633519 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:41:33.633497 2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Sep 10 13:41:33.633662 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:41:33.633609 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-wtblv\"" Sep 10 13:41:33.654213 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:41:33.654189 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-fa4d53-9hzcl"] Sep 10 13:41:33.761375 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:41:33.761346 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/572c821e-5dce-41c3-b895-4f60ca3089dc-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-b-fa4d53-9hzcl\" (UID: \"572c821e-5dce-41c3-b895-4f60ca3089dc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-fa4d53-9hzcl" Sep 10 13:41:33.862613 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:41:33.862589 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/572c821e-5dce-41c3-b895-4f60ca3089dc-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-b-fa4d53-9hzcl\" (UID: \"572c821e-5dce-41c3-b895-4f60ca3089dc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-fa4d53-9hzcl" Sep 10 13:41:33.863156 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:41:33.863138 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/572c821e-5dce-41c3-b895-4f60ca3089dc-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-b-fa4d53-9hzcl\" (UID: \"572c821e-5dce-41c3-b895-4f60ca3089dc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-fa4d53-9hzcl" Sep 10 13:41:33.941053 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:41:33.940969 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-fa4d53-9hzcl" Sep 10 13:41:34.221675 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:41:34.221640 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-fa4d53-9hzcl"] Sep 10 13:41:34.222142 ip-10-0-139-174 kubenswrapper[2557]: W0910 13:41:34.222115 2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod572c821e_5dce_41c3_b895_4f60ca3089dc.slice/crio-10c659b45ecb1e359872b1278fdf3a233661c5fe070a0c34b1f0c6caed1fb660 WatchSource:0}: Error finding container 10c659b45ecb1e359872b1278fdf3a233661c5fe070a0c34b1f0c6caed1fb660: Status 404 returned error can't find the container with id 10c659b45ecb1e359872b1278fdf3a233661c5fe070a0c34b1f0c6caed1fb660 Sep 10 13:41:34.764099 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:41:34.764006 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-fa4d53-9hzcl" event={"ID":"572c821e-5dce-41c3-b895-4f60ca3089dc","Type":"ContainerStarted","Data":"c501ba5ea4c4fb70a601ca75da4db8513f4451f6eb0ce1139b323a5fbc4e5ca5"} Sep 10 13:41:34.764099 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:41:34.764057 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-fa4d53-9hzcl" event={"ID":"572c821e-5dce-41c3-b895-4f60ca3089dc","Type":"ContainerStarted","Data":"10c659b45ecb1e359872b1278fdf3a233661c5fe070a0c34b1f0c6caed1fb660"} Sep 10 13:41:34.787203 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:41:34.787131 2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-fa4d53-9hzcl" podStartSLOduration=1.78711616 podStartE2EDuration="1.78711616s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 13:41:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 13:41:34.785206612 +0000 UTC m=+1455.092201025" watchObservedRunningTime="2026-09-10 13:41:34.78711616 +0000 UTC m=+1455.094110573" Sep 10 13:41:35.607457 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:41:35.607410 2557 patch_prober.go:28] interesting pod/e2e-api-a-559bdf-openshift-default-665458d45d-7lc67 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Sep 10 13:41:35.607624 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:41:35.607476 2557 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-a-559bdf-openshift-default-665458d45d-7lc67" podUID="cbf905c7-d254-490f-820e-83e29ff8b86d" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Sep 10 13:41:35.771763 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:41:35.771733 2557 generic.go:356] "Generic (PLEG): container finished" podID="572c821e-5dce-41c3-b895-4f60ca3089dc" containerID="c501ba5ea4c4fb70a601ca75da4db8513f4451f6eb0ce1139b323a5fbc4e5ca5" exitCode=0 Sep 10 13:41:35.772095 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:41:35.771812 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-fa4d53-9hzcl" event={"ID":"572c821e-5dce-41c3-b895-4f60ca3089dc","Type":"ContainerDied","Data":"c501ba5ea4c4fb70a601ca75da4db8513f4451f6eb0ce1139b323a5fbc4e5ca5"} Sep 10 13:41:36.908126 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:41:36.908100 2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-fa4d53-9hzcl" Sep 10 13:41:37.000772 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:41:37.000734 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/572c821e-5dce-41c3-b895-4f60ca3089dc-maas-api-service-ca\") pod \"572c821e-5dce-41c3-b895-4f60ca3089dc\" (UID: \"572c821e-5dce-41c3-b895-4f60ca3089dc\") " Sep 10 13:41:37.001115 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:41:37.001094 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/572c821e-5dce-41c3-b895-4f60ca3089dc-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "572c821e-5dce-41c3-b895-4f60ca3089dc" (UID: "572c821e-5dce-41c3-b895-4f60ca3089dc"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 10 13:41:37.001260 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:41:37.001243 2557 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/572c821e-5dce-41c3-b895-4f60ca3089dc-maas-api-service-ca\") on node \"ip-10-0-139-174.ec2.internal\" DevicePath \"\"" Sep 10 13:41:37.784904 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:41:37.784868 2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-fa4d53-9hzcl" Sep 10 13:41:37.785083 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:41:37.784914 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-fa4d53-9hzcl" event={"ID":"572c821e-5dce-41c3-b895-4f60ca3089dc","Type":"ContainerDied","Data":"10c659b45ecb1e359872b1278fdf3a233661c5fe070a0c34b1f0c6caed1fb660"} Sep 10 13:41:37.785083 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:41:37.784954 2557 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="10c659b45ecb1e359872b1278fdf3a233661c5fe070a0c34b1f0c6caed1fb660" Sep 10 13:41:38.489407 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:41:38.489385 2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-559bdf-openshift-default-665458d45d-7lc67" Sep 10 13:41:38.616160 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:41:38.616093 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cbf905c7-d254-490f-820e-83e29ff8b86d-istiod-ca-cert\") pod \"cbf905c7-d254-490f-820e-83e29ff8b86d\" (UID: \"cbf905c7-d254-490f-820e-83e29ff8b86d\") " Sep 10 13:41:38.616160 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:41:38.616141 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cbf905c7-d254-490f-820e-83e29ff8b86d-istio-envoy\") pod \"cbf905c7-d254-490f-820e-83e29ff8b86d\" (UID: \"cbf905c7-d254-490f-820e-83e29ff8b86d\") " Sep 10 13:41:38.616377 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:41:38.616173 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cbf905c7-d254-490f-820e-83e29ff8b86d-credential-socket\") pod \"cbf905c7-d254-490f-820e-83e29ff8b86d\" (UID: \"cbf905c7-d254-490f-820e-83e29ff8b86d\") " Sep 10 13:41:38.616377 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:41:38.616204 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cbf905c7-d254-490f-820e-83e29ff8b86d-workload-socket\") pod \"cbf905c7-d254-490f-820e-83e29ff8b86d\" (UID: \"cbf905c7-d254-490f-820e-83e29ff8b86d\") " Sep 10 13:41:38.616377 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:41:38.616253 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cbf905c7-d254-490f-820e-83e29ff8b86d-istio-token\") pod \"cbf905c7-d254-490f-820e-83e29ff8b86d\" (UID: \"cbf905c7-d254-490f-820e-83e29ff8b86d\") " Sep 10 13:41:38.616377 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:41:38.616279 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cbf905c7-d254-490f-820e-83e29ff8b86d-istio-podinfo\") pod \"cbf905c7-d254-490f-820e-83e29ff8b86d\" (UID: \"cbf905c7-d254-490f-820e-83e29ff8b86d\") " Sep 10 13:41:38.616377 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:41:38.616304 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cbf905c7-d254-490f-820e-83e29ff8b86d-istio-data\") pod \"cbf905c7-d254-490f-820e-83e29ff8b86d\" (UID: \"cbf905c7-d254-490f-820e-83e29ff8b86d\") " Sep 10 13:41:38.616377 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:41:38.616348 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qc8d4\" (UniqueName: \"kubernetes.io/projected/cbf905c7-d254-490f-820e-83e29ff8b86d-kube-api-access-qc8d4\") pod \"cbf905c7-d254-490f-820e-83e29ff8b86d\" (UID: \"cbf905c7-d254-490f-820e-83e29ff8b86d\") " Sep 10 13:41:38.616377 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:41:38.616370 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cbf905c7-d254-490f-820e-83e29ff8b86d-workload-certs\") pod \"cbf905c7-d254-490f-820e-83e29ff8b86d\" (UID: \"cbf905c7-d254-490f-820e-83e29ff8b86d\") " Sep 10 13:41:38.616748 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:41:38.616501 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cbf905c7-d254-490f-820e-83e29ff8b86d-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "cbf905c7-d254-490f-820e-83e29ff8b86d" (UID: "cbf905c7-d254-490f-820e-83e29ff8b86d"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 10 13:41:38.616748 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:41:38.616609 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cbf905c7-d254-490f-820e-83e29ff8b86d-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "cbf905c7-d254-490f-820e-83e29ff8b86d" (UID: "cbf905c7-d254-490f-820e-83e29ff8b86d"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 13:41:38.616748 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:41:38.616718 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cbf905c7-d254-490f-820e-83e29ff8b86d-istio-data" (OuterVolumeSpecName: "istio-data") pod "cbf905c7-d254-490f-820e-83e29ff8b86d" (UID: "cbf905c7-d254-490f-820e-83e29ff8b86d"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 13:41:38.617070 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:41:38.616807 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cbf905c7-d254-490f-820e-83e29ff8b86d-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "cbf905c7-d254-490f-820e-83e29ff8b86d" (UID: "cbf905c7-d254-490f-820e-83e29ff8b86d"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 13:41:38.617070 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:41:38.616788 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cbf905c7-d254-490f-820e-83e29ff8b86d-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "cbf905c7-d254-490f-820e-83e29ff8b86d" (UID: "cbf905c7-d254-490f-820e-83e29ff8b86d"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 13:41:38.617070 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:41:38.616827 2557 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cbf905c7-d254-490f-820e-83e29ff8b86d-istiod-ca-cert\") on node \"ip-10-0-139-174.ec2.internal\" DevicePath \"\"" Sep 10 13:41:38.617070 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:41:38.616847 2557 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cbf905c7-d254-490f-820e-83e29ff8b86d-credential-socket\") on node \"ip-10-0-139-174.ec2.internal\" DevicePath \"\"" Sep 10 13:41:38.618552 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:41:38.618521 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/cbf905c7-d254-490f-820e-83e29ff8b86d-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "cbf905c7-d254-490f-820e-83e29ff8b86d" (UID: "cbf905c7-d254-490f-820e-83e29ff8b86d"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Sep 10 13:41:38.618665 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:41:38.618560 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cbf905c7-d254-490f-820e-83e29ff8b86d-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "cbf905c7-d254-490f-820e-83e29ff8b86d" (UID: "cbf905c7-d254-490f-820e-83e29ff8b86d"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 13:41:38.618665 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:41:38.618605 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cbf905c7-d254-490f-820e-83e29ff8b86d-istio-token" (OuterVolumeSpecName: "istio-token") pod "cbf905c7-d254-490f-820e-83e29ff8b86d" (UID: "cbf905c7-d254-490f-820e-83e29ff8b86d"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 10 13:41:38.618665 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:41:38.618640 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cbf905c7-d254-490f-820e-83e29ff8b86d-kube-api-access-qc8d4" (OuterVolumeSpecName: "kube-api-access-qc8d4") pod "cbf905c7-d254-490f-820e-83e29ff8b86d" (UID: "cbf905c7-d254-490f-820e-83e29ff8b86d"). InnerVolumeSpecName "kube-api-access-qc8d4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 10 13:41:38.717955 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:41:38.717933 2557 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cbf905c7-d254-490f-820e-83e29ff8b86d-istio-token\") on node \"ip-10-0-139-174.ec2.internal\" DevicePath \"\"" Sep 10 13:41:38.717955 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:41:38.717955 2557 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cbf905c7-d254-490f-820e-83e29ff8b86d-istio-podinfo\") on node \"ip-10-0-139-174.ec2.internal\" DevicePath \"\"" Sep 10 13:41:38.718128 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:41:38.717963 2557 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cbf905c7-d254-490f-820e-83e29ff8b86d-istio-data\") on node \"ip-10-0-139-174.ec2.internal\" DevicePath \"\"" Sep 10 13:41:38.718128 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:41:38.717973 2557 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qc8d4\" (UniqueName: \"kubernetes.io/projected/cbf905c7-d254-490f-820e-83e29ff8b86d-kube-api-access-qc8d4\") on node \"ip-10-0-139-174.ec2.internal\" DevicePath \"\"" Sep 10 13:41:38.718128 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:41:38.717983 2557 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cbf905c7-d254-490f-820e-83e29ff8b86d-workload-certs\") on node \"ip-10-0-139-174.ec2.internal\" DevicePath \"\"" Sep 10 13:41:38.718128 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:41:38.717992 2557 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cbf905c7-d254-490f-820e-83e29ff8b86d-istio-envoy\") on node \"ip-10-0-139-174.ec2.internal\" DevicePath \"\"" Sep 10 13:41:38.718128 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:41:38.717999 2557 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cbf905c7-d254-490f-820e-83e29ff8b86d-workload-socket\") on node \"ip-10-0-139-174.ec2.internal\" DevicePath \"\"" Sep 10 13:41:38.793077 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:41:38.793054 2557 generic.go:356] "Generic (PLEG): container finished" podID="cbf905c7-d254-490f-820e-83e29ff8b86d" containerID="d7417d4f3718eea48c6971cf35d3fc1ee371f77f0a121d1d63150986913e6211" exitCode=0 Sep 10 13:41:38.793191 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:41:38.793109 2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-559bdf-openshift-default-665458d45d-7lc67" Sep 10 13:41:38.793191 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:41:38.793144 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-559bdf-openshift-default-665458d45d-7lc67" event={"ID":"cbf905c7-d254-490f-820e-83e29ff8b86d","Type":"ContainerDied","Data":"d7417d4f3718eea48c6971cf35d3fc1ee371f77f0a121d1d63150986913e6211"} Sep 10 13:41:38.793254 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:41:38.793192 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-559bdf-openshift-default-665458d45d-7lc67" event={"ID":"cbf905c7-d254-490f-820e-83e29ff8b86d","Type":"ContainerDied","Data":"d2a79a7d4c2e11a9ad68cdbcb0b44fafd249f03fe3c8823291aff91ef98ad9b9"} Sep 10 13:41:38.793254 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:41:38.793214 2557 scope.go:117] "RemoveContainer" containerID="d7417d4f3718eea48c6971cf35d3fc1ee371f77f0a121d1d63150986913e6211" Sep 10 13:41:38.813474 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:41:38.813449 2557 scope.go:117] "RemoveContainer" containerID="e2352722942044761ba03fe59fba912f2ac64597061392d075b001ac3073a016" Sep 10 13:41:38.822521 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:41:38.822498 2557 scope.go:117] "RemoveContainer" containerID="d7417d4f3718eea48c6971cf35d3fc1ee371f77f0a121d1d63150986913e6211" Sep 10 13:41:38.822834 ip-10-0-139-174 kubenswrapper[2557]: E0910 13:41:38.822815 2557 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d7417d4f3718eea48c6971cf35d3fc1ee371f77f0a121d1d63150986913e6211\": container with ID starting with d7417d4f3718eea48c6971cf35d3fc1ee371f77f0a121d1d63150986913e6211 not found: ID does not exist" containerID="d7417d4f3718eea48c6971cf35d3fc1ee371f77f0a121d1d63150986913e6211" Sep 10 13:41:38.822886 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:41:38.822847 2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d7417d4f3718eea48c6971cf35d3fc1ee371f77f0a121d1d63150986913e6211"} err="failed to get container status \"d7417d4f3718eea48c6971cf35d3fc1ee371f77f0a121d1d63150986913e6211\": rpc error: code = NotFound desc = could not find container \"d7417d4f3718eea48c6971cf35d3fc1ee371f77f0a121d1d63150986913e6211\": container with ID starting with d7417d4f3718eea48c6971cf35d3fc1ee371f77f0a121d1d63150986913e6211 not found: ID does not exist" Sep 10 13:41:38.822886 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:41:38.822867 2557 scope.go:117] "RemoveContainer" containerID="e2352722942044761ba03fe59fba912f2ac64597061392d075b001ac3073a016" Sep 10 13:41:38.823112 ip-10-0-139-174 kubenswrapper[2557]: E0910 13:41:38.823097 2557 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e2352722942044761ba03fe59fba912f2ac64597061392d075b001ac3073a016\": container with ID starting with e2352722942044761ba03fe59fba912f2ac64597061392d075b001ac3073a016 not found: ID does not exist" containerID="e2352722942044761ba03fe59fba912f2ac64597061392d075b001ac3073a016" Sep 10 13:41:38.823154 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:41:38.823117 2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e2352722942044761ba03fe59fba912f2ac64597061392d075b001ac3073a016"} err="failed to get container status \"e2352722942044761ba03fe59fba912f2ac64597061392d075b001ac3073a016\": rpc error: code = NotFound desc = could not find container \"e2352722942044761ba03fe59fba912f2ac64597061392d075b001ac3073a016\": container with ID starting with e2352722942044761ba03fe59fba912f2ac64597061392d075b001ac3073a016 not found: ID does not exist" Sep 10 13:41:38.823841 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:41:38.823821 2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-559bdf-openshift-default-665458d45d-7lc67"] Sep 10 13:41:38.832375 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:41:38.832357 2557 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-a-559bdf-openshift-default-665458d45d-7lc67"] Sep 10 13:41:40.220493 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:41:40.220457 2557 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="cbf905c7-d254-490f-820e-83e29ff8b86d" path="/var/lib/kubelet/pods/cbf905c7-d254-490f-820e-83e29ff8b86d/volumes" Sep 10 13:41:50.420169 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:41:50.420136 2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-fa4d53-9hzcl"] Sep 10 13:41:50.426392 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:41:50.426349 2557 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-fa4d53-9hzcl"] Sep 10 13:41:52.220252 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:41:52.220211 2557 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="572c821e-5dce-41c3-b895-4f60ca3089dc" path="/var/lib/kubelet/pods/572c821e-5dce-41c3-b895-4f60ca3089dc/volumes" Sep 10 13:41:58.835260 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:41:58.835227 2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-fa4d53-openshift-default-67c485f77b-l5b79"] Sep 10 13:41:58.835773 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:41:58.835474 2557 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-api-b-fa4d53-openshift-default-67c485f77b-l5b79" podUID="e39de022-9276-43ee-a39f-875f31e695a0" containerName="istio-proxy" containerID="cri-o://3884b5f75442f5aa3d831df266642716f1c0db41c71416a2478fb6401e70741a" gracePeriod=30 Sep 10 13:41:59.222607 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:41:59.222572 2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-c165a4-9wwc5"] Sep 10 13:41:59.222978 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:41:59.222967 2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cbf905c7-d254-490f-820e-83e29ff8b86d" containerName="istio-proxy" Sep 10 13:41:59.223024 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:41:59.222980 2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="cbf905c7-d254-490f-820e-83e29ff8b86d" containerName="istio-proxy" Sep 10 13:41:59.223024 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:41:59.222994 2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="572c821e-5dce-41c3-b895-4f60ca3089dc" containerName="revoke-keys" Sep 10 13:41:59.223024 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:41:59.222999 2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="572c821e-5dce-41c3-b895-4f60ca3089dc" containerName="revoke-keys" Sep 10 13:41:59.223024 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:41:59.223008 2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cbf905c7-d254-490f-820e-83e29ff8b86d" containerName="istio-proxy" Sep 10 13:41:59.223024 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:41:59.223014 2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="cbf905c7-d254-490f-820e-83e29ff8b86d" containerName="istio-proxy" Sep 10 13:41:59.223183 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:41:59.223082 2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="cbf905c7-d254-490f-820e-83e29ff8b86d" containerName="istio-proxy" Sep 10 13:41:59.223183 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:41:59.223089 2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="cbf905c7-d254-490f-820e-83e29ff8b86d" containerName="istio-proxy" Sep 10 13:41:59.223183 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:41:59.223097 2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="572c821e-5dce-41c3-b895-4f60ca3089dc" containerName="revoke-keys" Sep 10 13:41:59.225134 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:41:59.225117 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-c165a4-9wwc5" Sep 10 13:41:59.229013 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:41:59.228994 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-wtblv\"" Sep 10 13:41:59.229145 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:41:59.228995 2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Sep 10 13:41:59.235964 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:41:59.235939 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-c165a4-9wwc5"] Sep 10 13:41:59.334511 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:41:59.334466 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/68d47d5d-1673-48df-b01a-1b83153dfbb3-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w2-c165a4-9wwc5\" (UID: \"68d47d5d-1673-48df-b01a-1b83153dfbb3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-c165a4-9wwc5" Sep 10 13:41:59.435689 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:41:59.435663 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/68d47d5d-1673-48df-b01a-1b83153dfbb3-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w2-c165a4-9wwc5\" (UID: \"68d47d5d-1673-48df-b01a-1b83153dfbb3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-c165a4-9wwc5" Sep 10 13:41:59.436242 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:41:59.436216 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/68d47d5d-1673-48df-b01a-1b83153dfbb3-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w2-c165a4-9wwc5\" (UID: \"68d47d5d-1673-48df-b01a-1b83153dfbb3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-c165a4-9wwc5" Sep 10 13:41:59.536556 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:41:59.536464 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-c165a4-9wwc5" Sep 10 13:41:59.806955 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:41:59.806924 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-c165a4-9wwc5"] Sep 10 13:41:59.808458 ip-10-0-139-174 kubenswrapper[2557]: W0910 13:41:59.808401 2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod68d47d5d_1673_48df_b01a_1b83153dfbb3.slice/crio-dfcd07008a4b9a683a05a28c7bab4716cbe873ee26508b45ac425cf9e3161c10 WatchSource:0}: Error finding container dfcd07008a4b9a683a05a28c7bab4716cbe873ee26508b45ac425cf9e3161c10: Status 404 returned error can't find the container with id dfcd07008a4b9a683a05a28c7bab4716cbe873ee26508b45ac425cf9e3161c10 Sep 10 13:41:59.919899 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:41:59.919870 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-c165a4-9wwc5" event={"ID":"68d47d5d-1673-48df-b01a-1b83153dfbb3","Type":"ContainerStarted","Data":"dfcd07008a4b9a683a05a28c7bab4716cbe873ee26508b45ac425cf9e3161c10"} Sep 10 13:42:00.927349 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:42:00.927260 2557 generic.go:356] "Generic (PLEG): container finished" podID="68d47d5d-1673-48df-b01a-1b83153dfbb3" containerID="04c9e6c2e58e0b8fb6ae0bbad11a4dd4aa208b6c9f287ed6d8dfef7008512a07" exitCode=0 Sep 10 13:42:00.927786 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:42:00.927344 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-c165a4-9wwc5" event={"ID":"68d47d5d-1673-48df-b01a-1b83153dfbb3","Type":"ContainerDied","Data":"04c9e6c2e58e0b8fb6ae0bbad11a4dd4aa208b6c9f287ed6d8dfef7008512a07"} Sep 10 13:42:02.092306 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:42:02.091807 2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-c165a4-9wwc5" Sep 10 13:42:02.265197 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:42:02.265168 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/68d47d5d-1673-48df-b01a-1b83153dfbb3-maas-api-service-ca\") pod \"68d47d5d-1673-48df-b01a-1b83153dfbb3\" (UID: \"68d47d5d-1673-48df-b01a-1b83153dfbb3\") " Sep 10 13:42:02.265524 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:42:02.265501 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/68d47d5d-1673-48df-b01a-1b83153dfbb3-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "68d47d5d-1673-48df-b01a-1b83153dfbb3" (UID: "68d47d5d-1673-48df-b01a-1b83153dfbb3"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 10 13:42:02.265614 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:42:02.265594 2557 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/68d47d5d-1673-48df-b01a-1b83153dfbb3-maas-api-service-ca\") on node \"ip-10-0-139-174.ec2.internal\" DevicePath \"\"" Sep 10 13:42:02.941268 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:42:02.941243 2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-c165a4-9wwc5" Sep 10 13:42:02.941268 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:42:02.941259 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-c165a4-9wwc5" event={"ID":"68d47d5d-1673-48df-b01a-1b83153dfbb3","Type":"ContainerDied","Data":"dfcd07008a4b9a683a05a28c7bab4716cbe873ee26508b45ac425cf9e3161c10"} Sep 10 13:42:02.941488 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:42:02.941306 2557 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="dfcd07008a4b9a683a05a28c7bab4716cbe873ee26508b45ac425cf9e3161c10" Sep 10 13:42:04.092594 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:42:04.092564 2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-fa4d53-openshift-default-67c485f77b-l5b79" Sep 10 13:42:04.183791 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:42:04.183706 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e39de022-9276-43ee-a39f-875f31e695a0-credential-socket\") pod \"e39de022-9276-43ee-a39f-875f31e695a0\" (UID: \"e39de022-9276-43ee-a39f-875f31e695a0\") " Sep 10 13:42:04.183791 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:42:04.183768 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e39de022-9276-43ee-a39f-875f31e695a0-istio-data\") pod \"e39de022-9276-43ee-a39f-875f31e695a0\" (UID: \"e39de022-9276-43ee-a39f-875f31e695a0\") " Sep 10 13:42:04.183791 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:42:04.183785 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e39de022-9276-43ee-a39f-875f31e695a0-workload-socket\") pod \"e39de022-9276-43ee-a39f-875f31e695a0\" (UID: \"e39de022-9276-43ee-a39f-875f31e695a0\") " Sep 10 13:42:04.184042 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:42:04.183816 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e39de022-9276-43ee-a39f-875f31e695a0-workload-certs\") pod \"e39de022-9276-43ee-a39f-875f31e695a0\" (UID: \"e39de022-9276-43ee-a39f-875f31e695a0\") " Sep 10 13:42:04.184042 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:42:04.183840 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e39de022-9276-43ee-a39f-875f31e695a0-istio-podinfo\") pod \"e39de022-9276-43ee-a39f-875f31e695a0\" (UID: \"e39de022-9276-43ee-a39f-875f31e695a0\") " Sep 10 13:42:04.184042 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:42:04.183872 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6xpzm\" (UniqueName: \"kubernetes.io/projected/e39de022-9276-43ee-a39f-875f31e695a0-kube-api-access-6xpzm\") pod \"e39de022-9276-43ee-a39f-875f31e695a0\" (UID: \"e39de022-9276-43ee-a39f-875f31e695a0\") " Sep 10 13:42:04.184042 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:42:04.183895 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e39de022-9276-43ee-a39f-875f31e695a0-istio-envoy\") pod \"e39de022-9276-43ee-a39f-875f31e695a0\" (UID: \"e39de022-9276-43ee-a39f-875f31e695a0\") " Sep 10 13:42:04.184042 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:42:04.184002 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e39de022-9276-43ee-a39f-875f31e695a0-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "e39de022-9276-43ee-a39f-875f31e695a0" (UID: "e39de022-9276-43ee-a39f-875f31e695a0"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 13:42:04.184042 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:42:04.184017 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e39de022-9276-43ee-a39f-875f31e695a0-istio-data" (OuterVolumeSpecName: "istio-data") pod "e39de022-9276-43ee-a39f-875f31e695a0" (UID: "e39de022-9276-43ee-a39f-875f31e695a0"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 13:42:04.184042 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:42:04.184037 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e39de022-9276-43ee-a39f-875f31e695a0-istio-token\") pod \"e39de022-9276-43ee-a39f-875f31e695a0\" (UID: \"e39de022-9276-43ee-a39f-875f31e695a0\") " Sep 10 13:42:04.184386 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:42:04.184064 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e39de022-9276-43ee-a39f-875f31e695a0-istiod-ca-cert\") pod \"e39de022-9276-43ee-a39f-875f31e695a0\" (UID: \"e39de022-9276-43ee-a39f-875f31e695a0\") " Sep 10 13:42:04.184386 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:42:04.184152 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e39de022-9276-43ee-a39f-875f31e695a0-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "e39de022-9276-43ee-a39f-875f31e695a0" (UID: "e39de022-9276-43ee-a39f-875f31e695a0"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 13:42:04.184386 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:42:04.184197 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e39de022-9276-43ee-a39f-875f31e695a0-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "e39de022-9276-43ee-a39f-875f31e695a0" (UID: "e39de022-9276-43ee-a39f-875f31e695a0"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 13:42:04.184580 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:42:04.184483 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e39de022-9276-43ee-a39f-875f31e695a0-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "e39de022-9276-43ee-a39f-875f31e695a0" (UID: "e39de022-9276-43ee-a39f-875f31e695a0"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 10 13:42:04.184580 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:42:04.184567 2557 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e39de022-9276-43ee-a39f-875f31e695a0-credential-socket\") on node \"ip-10-0-139-174.ec2.internal\" DevicePath \"\"" Sep 10 13:42:04.184850 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:42:04.184584 2557 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e39de022-9276-43ee-a39f-875f31e695a0-istio-data\") on node \"ip-10-0-139-174.ec2.internal\" DevicePath \"\"" Sep 10 13:42:04.184850 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:42:04.184597 2557 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e39de022-9276-43ee-a39f-875f31e695a0-workload-socket\") on node \"ip-10-0-139-174.ec2.internal\" DevicePath \"\"" Sep 10 13:42:04.184850 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:42:04.184611 2557 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e39de022-9276-43ee-a39f-875f31e695a0-workload-certs\") on node \"ip-10-0-139-174.ec2.internal\" DevicePath \"\"" Sep 10 13:42:04.184850 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:42:04.184621 2557 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e39de022-9276-43ee-a39f-875f31e695a0-istiod-ca-cert\") on node \"ip-10-0-139-174.ec2.internal\" DevicePath \"\"" Sep 10 13:42:04.186230 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:42:04.186201 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e39de022-9276-43ee-a39f-875f31e695a0-istio-token" (OuterVolumeSpecName: "istio-token") pod "e39de022-9276-43ee-a39f-875f31e695a0" (UID: "e39de022-9276-43ee-a39f-875f31e695a0"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 10 13:42:04.186351 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:42:04.186300 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e39de022-9276-43ee-a39f-875f31e695a0-kube-api-access-6xpzm" (OuterVolumeSpecName: "kube-api-access-6xpzm") pod "e39de022-9276-43ee-a39f-875f31e695a0" (UID: "e39de022-9276-43ee-a39f-875f31e695a0"). InnerVolumeSpecName "kube-api-access-6xpzm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 10 13:42:04.186685 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:42:04.186662 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e39de022-9276-43ee-a39f-875f31e695a0-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "e39de022-9276-43ee-a39f-875f31e695a0" (UID: "e39de022-9276-43ee-a39f-875f31e695a0"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 13:42:04.186764 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:42:04.186678 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/e39de022-9276-43ee-a39f-875f31e695a0-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "e39de022-9276-43ee-a39f-875f31e695a0" (UID: "e39de022-9276-43ee-a39f-875f31e695a0"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Sep 10 13:42:04.285881 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:42:04.285858 2557 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e39de022-9276-43ee-a39f-875f31e695a0-istio-podinfo\") on node \"ip-10-0-139-174.ec2.internal\" DevicePath \"\"" Sep 10 13:42:04.285881 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:42:04.285879 2557 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6xpzm\" (UniqueName: \"kubernetes.io/projected/e39de022-9276-43ee-a39f-875f31e695a0-kube-api-access-6xpzm\") on node \"ip-10-0-139-174.ec2.internal\" DevicePath \"\"" Sep 10 13:42:04.286082 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:42:04.285890 2557 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e39de022-9276-43ee-a39f-875f31e695a0-istio-envoy\") on node \"ip-10-0-139-174.ec2.internal\" DevicePath \"\"" Sep 10 13:42:04.286082 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:42:04.285898 2557 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e39de022-9276-43ee-a39f-875f31e695a0-istio-token\") on node \"ip-10-0-139-174.ec2.internal\" DevicePath \"\"" Sep 10 13:42:04.956969 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:42:04.956937 2557 generic.go:356] "Generic (PLEG): container finished" podID="e39de022-9276-43ee-a39f-875f31e695a0" containerID="3884b5f75442f5aa3d831df266642716f1c0db41c71416a2478fb6401e70741a" exitCode=0 Sep 10 13:42:04.957125 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:42:04.956989 2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-fa4d53-openshift-default-67c485f77b-l5b79" Sep 10 13:42:04.957125 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:42:04.956993 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-fa4d53-openshift-default-67c485f77b-l5b79" event={"ID":"e39de022-9276-43ee-a39f-875f31e695a0","Type":"ContainerDied","Data":"3884b5f75442f5aa3d831df266642716f1c0db41c71416a2478fb6401e70741a"} Sep 10 13:42:04.957125 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:42:04.957034 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-fa4d53-openshift-default-67c485f77b-l5b79" event={"ID":"e39de022-9276-43ee-a39f-875f31e695a0","Type":"ContainerDied","Data":"33c094c7fedbf42054ffd4bf2f3982c3838ec6d668e21a8db9cc7cd02bbff8f9"} Sep 10 13:42:04.957125 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:42:04.957054 2557 scope.go:117] "RemoveContainer" containerID="3884b5f75442f5aa3d831df266642716f1c0db41c71416a2478fb6401e70741a" Sep 10 13:42:04.973236 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:42:04.973216 2557 scope.go:117] "RemoveContainer" containerID="ea17cc3f405ab5366dd61861c8d5f4c92c56e5de8eda6382c26e2a5d595e91c4" Sep 10 13:42:04.982245 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:42:04.982222 2557 scope.go:117] "RemoveContainer" containerID="3884b5f75442f5aa3d831df266642716f1c0db41c71416a2478fb6401e70741a" Sep 10 13:42:04.982524 ip-10-0-139-174 kubenswrapper[2557]: E0910 13:42:04.982498 2557 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3884b5f75442f5aa3d831df266642716f1c0db41c71416a2478fb6401e70741a\": container with ID starting with 3884b5f75442f5aa3d831df266642716f1c0db41c71416a2478fb6401e70741a not found: ID does not exist" containerID="3884b5f75442f5aa3d831df266642716f1c0db41c71416a2478fb6401e70741a" Sep 10 13:42:04.982600 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:42:04.982532 2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3884b5f75442f5aa3d831df266642716f1c0db41c71416a2478fb6401e70741a"} err="failed to get container status \"3884b5f75442f5aa3d831df266642716f1c0db41c71416a2478fb6401e70741a\": rpc error: code = NotFound desc = could not find container \"3884b5f75442f5aa3d831df266642716f1c0db41c71416a2478fb6401e70741a\": container with ID starting with 3884b5f75442f5aa3d831df266642716f1c0db41c71416a2478fb6401e70741a not found: ID does not exist" Sep 10 13:42:04.982600 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:42:04.982549 2557 scope.go:117] "RemoveContainer" containerID="ea17cc3f405ab5366dd61861c8d5f4c92c56e5de8eda6382c26e2a5d595e91c4" Sep 10 13:42:04.982794 ip-10-0-139-174 kubenswrapper[2557]: E0910 13:42:04.982778 2557 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ea17cc3f405ab5366dd61861c8d5f4c92c56e5de8eda6382c26e2a5d595e91c4\": container with ID starting with ea17cc3f405ab5366dd61861c8d5f4c92c56e5de8eda6382c26e2a5d595e91c4 not found: ID does not exist" containerID="ea17cc3f405ab5366dd61861c8d5f4c92c56e5de8eda6382c26e2a5d595e91c4" Sep 10 13:42:04.982831 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:42:04.982800 2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ea17cc3f405ab5366dd61861c8d5f4c92c56e5de8eda6382c26e2a5d595e91c4"} err="failed to get container status \"ea17cc3f405ab5366dd61861c8d5f4c92c56e5de8eda6382c26e2a5d595e91c4\": rpc error: code = NotFound desc = could not find container \"ea17cc3f405ab5366dd61861c8d5f4c92c56e5de8eda6382c26e2a5d595e91c4\": container with ID starting with ea17cc3f405ab5366dd61861c8d5f4c92c56e5de8eda6382c26e2a5d595e91c4 not found: ID does not exist" Sep 10 13:42:04.989797 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:42:04.989773 2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-fa4d53-openshift-default-67c485f77b-l5b79"] Sep 10 13:42:04.992219 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:42:04.992200 2557 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-b-fa4d53-openshift-default-67c485f77b-l5b79"] Sep 10 13:42:06.220101 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:42:06.220066 2557 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e39de022-9276-43ee-a39f-875f31e695a0" path="/var/lib/kubelet/pods/e39de022-9276-43ee-a39f-875f31e695a0/volumes" Sep 10 13:42:10.419327 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:42:10.419292 2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-c165a4-9wwc5"] Sep 10 13:42:10.424870 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:42:10.424846 2557 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-c165a4-9wwc5"] Sep 10 13:42:12.219485 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:42:12.219453 2557 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="68d47d5d-1673-48df-b01a-1b83153dfbb3" path="/var/lib/kubelet/pods/68d47d5d-1673-48df-b01a-1b83153dfbb3/volumes" Sep 10 13:42:18.549822 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:42:18.549781 2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-c165a4-openshift-default-5b5c6d994b-g4wbc"] Sep 10 13:42:18.550280 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:42:18.550142 2557 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w2-c165a4-openshift-default-5b5c6d994b-g4wbc" podUID="b65e81d0-d6df-44d2-a87a-04b4a58eb154" containerName="istio-proxy" containerID="cri-o://c1553bb4051824e55bf4fb30a37e64d5f2f738c8ced98dbc30e6ce76c2251d75" gracePeriod=30 Sep 10 13:42:20.270094 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:42:20.270064 2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-92n4q_720acda6-0b15-49ac-ab4e-60de70c4324a/ovn-acl-logging/0.log" Sep 10 13:42:20.279073 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:42:20.279053 2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-92n4q_720acda6-0b15-49ac-ab4e-60de70c4324a/ovn-acl-logging/0.log" Sep 10 13:42:22.209169 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:42:22.209135 2557 patch_prober.go:28] interesting pod/e2e-shared-a-w2-c165a4-openshift-default-5b5c6d994b-g4wbc container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Sep 10 13:42:22.209535 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:42:22.209202 2557 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-a-w2-c165a4-openshift-default-5b5c6d994b-g4wbc" podUID="b65e81d0-d6df-44d2-a87a-04b4a58eb154" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Sep 10 13:42:23.810782 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:42:23.810760 2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-c165a4-openshift-default-5b5c6d994b-g4wbc" Sep 10 13:42:23.860925 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:42:23.860850 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b65e81d0-d6df-44d2-a87a-04b4a58eb154-workload-socket\") pod \"b65e81d0-d6df-44d2-a87a-04b4a58eb154\" (UID: \"b65e81d0-d6df-44d2-a87a-04b4a58eb154\") " Sep 10 13:42:23.860925 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:42:23.860899 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b65e81d0-d6df-44d2-a87a-04b4a58eb154-istiod-ca-cert\") pod \"b65e81d0-d6df-44d2-a87a-04b4a58eb154\" (UID: \"b65e81d0-d6df-44d2-a87a-04b4a58eb154\") " Sep 10 13:42:23.861113 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:42:23.860928 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b65e81d0-d6df-44d2-a87a-04b4a58eb154-istio-token\") pod \"b65e81d0-d6df-44d2-a87a-04b4a58eb154\" (UID: \"b65e81d0-d6df-44d2-a87a-04b4a58eb154\") " Sep 10 13:42:23.861113 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:42:23.861051 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b65e81d0-d6df-44d2-a87a-04b4a58eb154-credential-socket\") pod \"b65e81d0-d6df-44d2-a87a-04b4a58eb154\" (UID: \"b65e81d0-d6df-44d2-a87a-04b4a58eb154\") " Sep 10 13:42:23.861113 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:42:23.861090 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b65e81d0-d6df-44d2-a87a-04b4a58eb154-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "b65e81d0-d6df-44d2-a87a-04b4a58eb154" (UID: "b65e81d0-d6df-44d2-a87a-04b4a58eb154"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 13:42:23.861113 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:42:23.861095 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b65e81d0-d6df-44d2-a87a-04b4a58eb154-istio-envoy\") pod \"b65e81d0-d6df-44d2-a87a-04b4a58eb154\" (UID: \"b65e81d0-d6df-44d2-a87a-04b4a58eb154\") " Sep 10 13:42:23.861324 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:42:23.861143 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2gvt2\" (UniqueName: \"kubernetes.io/projected/b65e81d0-d6df-44d2-a87a-04b4a58eb154-kube-api-access-2gvt2\") pod \"b65e81d0-d6df-44d2-a87a-04b4a58eb154\" (UID: \"b65e81d0-d6df-44d2-a87a-04b4a58eb154\") " Sep 10 13:42:23.861324 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:42:23.861211 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b65e81d0-d6df-44d2-a87a-04b4a58eb154-workload-certs\") pod \"b65e81d0-d6df-44d2-a87a-04b4a58eb154\" (UID: \"b65e81d0-d6df-44d2-a87a-04b4a58eb154\") " Sep 10 13:42:23.861324 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:42:23.861241 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b65e81d0-d6df-44d2-a87a-04b4a58eb154-istio-podinfo\") pod \"b65e81d0-d6df-44d2-a87a-04b4a58eb154\" (UID: \"b65e81d0-d6df-44d2-a87a-04b4a58eb154\") " Sep 10 13:42:23.861324 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:42:23.861268 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b65e81d0-d6df-44d2-a87a-04b4a58eb154-istio-data\") pod \"b65e81d0-d6df-44d2-a87a-04b4a58eb154\" (UID: \"b65e81d0-d6df-44d2-a87a-04b4a58eb154\") " Sep 10 13:42:23.861324 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:42:23.861292 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b65e81d0-d6df-44d2-a87a-04b4a58eb154-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "b65e81d0-d6df-44d2-a87a-04b4a58eb154" (UID: "b65e81d0-d6df-44d2-a87a-04b4a58eb154"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 13:42:23.861617 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:42:23.861336 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b65e81d0-d6df-44d2-a87a-04b4a58eb154-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "b65e81d0-d6df-44d2-a87a-04b4a58eb154" (UID: "b65e81d0-d6df-44d2-a87a-04b4a58eb154"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 10 13:42:23.861617 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:42:23.861526 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b65e81d0-d6df-44d2-a87a-04b4a58eb154-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "b65e81d0-d6df-44d2-a87a-04b4a58eb154" (UID: "b65e81d0-d6df-44d2-a87a-04b4a58eb154"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 13:42:23.861887 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:42:23.861707 2557 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b65e81d0-d6df-44d2-a87a-04b4a58eb154-credential-socket\") on node \"ip-10-0-139-174.ec2.internal\" DevicePath \"\"" Sep 10 13:42:23.861887 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:42:23.861726 2557 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b65e81d0-d6df-44d2-a87a-04b4a58eb154-workload-certs\") on node \"ip-10-0-139-174.ec2.internal\" DevicePath \"\"" Sep 10 13:42:23.861887 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:42:23.861740 2557 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b65e81d0-d6df-44d2-a87a-04b4a58eb154-workload-socket\") on node \"ip-10-0-139-174.ec2.internal\" DevicePath \"\"" Sep 10 13:42:23.861887 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:42:23.861752 2557 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b65e81d0-d6df-44d2-a87a-04b4a58eb154-istiod-ca-cert\") on node \"ip-10-0-139-174.ec2.internal\" DevicePath \"\"" Sep 10 13:42:23.861887 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:42:23.861857 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b65e81d0-d6df-44d2-a87a-04b4a58eb154-istio-data" (OuterVolumeSpecName: "istio-data") pod "b65e81d0-d6df-44d2-a87a-04b4a58eb154" (UID: "b65e81d0-d6df-44d2-a87a-04b4a58eb154"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 13:42:23.863898 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:42:23.863872 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b65e81d0-d6df-44d2-a87a-04b4a58eb154-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "b65e81d0-d6df-44d2-a87a-04b4a58eb154" (UID: "b65e81d0-d6df-44d2-a87a-04b4a58eb154"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 13:42:23.863898 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:42:23.863893 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b65e81d0-d6df-44d2-a87a-04b4a58eb154-istio-token" (OuterVolumeSpecName: "istio-token") pod "b65e81d0-d6df-44d2-a87a-04b4a58eb154" (UID: "b65e81d0-d6df-44d2-a87a-04b4a58eb154"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 10 13:42:23.864052 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:42:23.863880 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/b65e81d0-d6df-44d2-a87a-04b4a58eb154-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "b65e81d0-d6df-44d2-a87a-04b4a58eb154" (UID: "b65e81d0-d6df-44d2-a87a-04b4a58eb154"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Sep 10 13:42:23.864052 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:42:23.863919 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b65e81d0-d6df-44d2-a87a-04b4a58eb154-kube-api-access-2gvt2" (OuterVolumeSpecName: "kube-api-access-2gvt2") pod "b65e81d0-d6df-44d2-a87a-04b4a58eb154" (UID: "b65e81d0-d6df-44d2-a87a-04b4a58eb154"). InnerVolumeSpecName "kube-api-access-2gvt2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 10 13:42:23.962674 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:42:23.962651 2557 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b65e81d0-d6df-44d2-a87a-04b4a58eb154-istio-podinfo\") on node \"ip-10-0-139-174.ec2.internal\" DevicePath \"\"" Sep 10 13:42:23.962674 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:42:23.962671 2557 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b65e81d0-d6df-44d2-a87a-04b4a58eb154-istio-data\") on node \"ip-10-0-139-174.ec2.internal\" DevicePath \"\"" Sep 10 13:42:23.962674 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:42:23.962680 2557 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b65e81d0-d6df-44d2-a87a-04b4a58eb154-istio-token\") on node \"ip-10-0-139-174.ec2.internal\" DevicePath \"\"" Sep 10 13:42:23.962855 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:42:23.962688 2557 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b65e81d0-d6df-44d2-a87a-04b4a58eb154-istio-envoy\") on node \"ip-10-0-139-174.ec2.internal\" DevicePath \"\"" Sep 10 13:42:23.962855 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:42:23.962695 2557 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2gvt2\" (UniqueName: \"kubernetes.io/projected/b65e81d0-d6df-44d2-a87a-04b4a58eb154-kube-api-access-2gvt2\") on node \"ip-10-0-139-174.ec2.internal\" DevicePath \"\"" Sep 10 13:42:24.069316 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:42:24.069286 2557 generic.go:356] "Generic (PLEG): container finished" podID="b65e81d0-d6df-44d2-a87a-04b4a58eb154" containerID="c1553bb4051824e55bf4fb30a37e64d5f2f738c8ced98dbc30e6ce76c2251d75" exitCode=0 Sep 10 13:42:24.069484 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:42:24.069331 2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-c165a4-openshift-default-5b5c6d994b-g4wbc" Sep 10 13:42:24.069484 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:42:24.069372 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-c165a4-openshift-default-5b5c6d994b-g4wbc" event={"ID":"b65e81d0-d6df-44d2-a87a-04b4a58eb154","Type":"ContainerDied","Data":"c1553bb4051824e55bf4fb30a37e64d5f2f738c8ced98dbc30e6ce76c2251d75"} Sep 10 13:42:24.069484 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:42:24.069443 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-c165a4-openshift-default-5b5c6d994b-g4wbc" event={"ID":"b65e81d0-d6df-44d2-a87a-04b4a58eb154","Type":"ContainerDied","Data":"87ce13f9691faf7d7e29c596867d46edb83cf1387e6fbaa6a516ffe4a55547c8"} Sep 10 13:42:24.069484 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:42:24.069468 2557 scope.go:117] "RemoveContainer" containerID="c1553bb4051824e55bf4fb30a37e64d5f2f738c8ced98dbc30e6ce76c2251d75" Sep 10 13:42:24.079807 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:42:24.079792 2557 scope.go:117] "RemoveContainer" containerID="2cb49671b88e8282f7c5caa1f94efe48957c68fea70fb978a62b9c0445a20c56" Sep 10 13:42:24.088785 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:42:24.088764 2557 scope.go:117] "RemoveContainer" containerID="c1553bb4051824e55bf4fb30a37e64d5f2f738c8ced98dbc30e6ce76c2251d75" Sep 10 13:42:24.089100 ip-10-0-139-174 kubenswrapper[2557]: E0910 13:42:24.089081 2557 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c1553bb4051824e55bf4fb30a37e64d5f2f738c8ced98dbc30e6ce76c2251d75\": container with ID starting with c1553bb4051824e55bf4fb30a37e64d5f2f738c8ced98dbc30e6ce76c2251d75 not found: ID does not exist" containerID="c1553bb4051824e55bf4fb30a37e64d5f2f738c8ced98dbc30e6ce76c2251d75" Sep 10 13:42:24.089165 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:42:24.089105 2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c1553bb4051824e55bf4fb30a37e64d5f2f738c8ced98dbc30e6ce76c2251d75"} err="failed to get container status \"c1553bb4051824e55bf4fb30a37e64d5f2f738c8ced98dbc30e6ce76c2251d75\": rpc error: code = NotFound desc = could not find container \"c1553bb4051824e55bf4fb30a37e64d5f2f738c8ced98dbc30e6ce76c2251d75\": container with ID starting with c1553bb4051824e55bf4fb30a37e64d5f2f738c8ced98dbc30e6ce76c2251d75 not found: ID does not exist" Sep 10 13:42:24.089165 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:42:24.089121 2557 scope.go:117] "RemoveContainer" containerID="2cb49671b88e8282f7c5caa1f94efe48957c68fea70fb978a62b9c0445a20c56" Sep 10 13:42:24.089359 ip-10-0-139-174 kubenswrapper[2557]: E0910 13:42:24.089342 2557 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2cb49671b88e8282f7c5caa1f94efe48957c68fea70fb978a62b9c0445a20c56\": container with ID starting with 2cb49671b88e8282f7c5caa1f94efe48957c68fea70fb978a62b9c0445a20c56 not found: ID does not exist" containerID="2cb49671b88e8282f7c5caa1f94efe48957c68fea70fb978a62b9c0445a20c56" Sep 10 13:42:24.089409 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:42:24.089363 2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2cb49671b88e8282f7c5caa1f94efe48957c68fea70fb978a62b9c0445a20c56"} err="failed to get container status \"2cb49671b88e8282f7c5caa1f94efe48957c68fea70fb978a62b9c0445a20c56\": rpc error: code = NotFound desc = could not find container \"2cb49671b88e8282f7c5caa1f94efe48957c68fea70fb978a62b9c0445a20c56\": container with ID starting with 2cb49671b88e8282f7c5caa1f94efe48957c68fea70fb978a62b9c0445a20c56 not found: ID does not exist" Sep 10 13:42:24.103161 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:42:24.103129 2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-c165a4-openshift-default-5b5c6d994b-g4wbc"] Sep 10 13:42:24.107374 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:42:24.107355 2557 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-c165a4-openshift-default-5b5c6d994b-g4wbc"] Sep 10 13:42:24.113851 ip-10-0-139-174 kubenswrapper[2557]: E0910 13:42:24.113801 2557 cadvisor_stats_provider.go:567] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb65e81d0_d6df_44d2_a87a_04b4a58eb154.slice/crio-87ce13f9691faf7d7e29c596867d46edb83cf1387e6fbaa6a516ffe4a55547c8\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb65e81d0_d6df_44d2_a87a_04b4a58eb154.slice\": RecentStats: unable to find data in memory cache]" Sep 10 13:42:24.220230 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:42:24.220205 2557 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b65e81d0-d6df-44d2-a87a-04b4a58eb154" path="/var/lib/kubelet/pods/b65e81d0-d6df-44d2-a87a-04b4a58eb154/volumes" Sep 10 13:42:28.662312 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:42:28.662238 2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-08b85c-5cc85974f5-ncbzk"] Sep 10 13:42:28.663777 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:42:28.663749 2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e39de022-9276-43ee-a39f-875f31e695a0" containerName="istio-proxy" Sep 10 13:42:28.663777 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:42:28.663777 2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="e39de022-9276-43ee-a39f-875f31e695a0" containerName="istio-proxy" Sep 10 13:42:28.663985 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:42:28.663810 2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b65e81d0-d6df-44d2-a87a-04b4a58eb154" containerName="istio-proxy" Sep 10 13:42:28.663985 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:42:28.663819 2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="b65e81d0-d6df-44d2-a87a-04b4a58eb154" containerName="istio-proxy" Sep 10 13:42:28.663985 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:42:28.663839 2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b65e81d0-d6df-44d2-a87a-04b4a58eb154" containerName="istio-proxy" Sep 10 13:42:28.663985 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:42:28.663847 2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="b65e81d0-d6df-44d2-a87a-04b4a58eb154" containerName="istio-proxy" Sep 10 13:42:28.663985 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:42:28.663860 2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e39de022-9276-43ee-a39f-875f31e695a0" containerName="istio-proxy" Sep 10 13:42:28.663985 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:42:28.663868 2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="e39de022-9276-43ee-a39f-875f31e695a0" containerName="istio-proxy" Sep 10 13:42:28.663985 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:42:28.663882 2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="68d47d5d-1673-48df-b01a-1b83153dfbb3" containerName="revoke-keys" Sep 10 13:42:28.663985 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:42:28.663889 2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="68d47d5d-1673-48df-b01a-1b83153dfbb3" containerName="revoke-keys" Sep 10 13:42:28.664363 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:42:28.663995 2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="68d47d5d-1673-48df-b01a-1b83153dfbb3" containerName="revoke-keys" Sep 10 13:42:28.664363 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:42:28.664009 2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="b65e81d0-d6df-44d2-a87a-04b4a58eb154" containerName="istio-proxy" Sep 10 13:42:28.664363 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:42:28.664018 2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="e39de022-9276-43ee-a39f-875f31e695a0" containerName="istio-proxy" Sep 10 13:42:28.664363 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:42:28.664028 2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="e39de022-9276-43ee-a39f-875f31e695a0" containerName="istio-proxy" Sep 10 13:42:28.664363 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:42:28.664037 2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="b65e81d0-d6df-44d2-a87a-04b4a58eb154" containerName="istio-proxy" Sep 10 13:42:28.669150 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:42:28.669121 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-08b85c-5cc85974f5-ncbzk" Sep 10 13:42:28.674798 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:42:28.674205 2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-b-08b85c\"" Sep 10 13:42:28.710252 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:42:28.710223 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f676c0de-5f1d-4ebc-aa86-6de59306b195-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-08b85c-5cc85974f5-ncbzk\" (UID: \"f676c0de-5f1d-4ebc-aa86-6de59306b195\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-08b85c-5cc85974f5-ncbzk" Sep 10 13:42:28.710502 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:42:28.710384 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tqwvj\" (UniqueName: \"kubernetes.io/projected/f676c0de-5f1d-4ebc-aa86-6de59306b195-kube-api-access-tqwvj\") pod \"payload-processing-e2e-ipp-b-08b85c-5cc85974f5-ncbzk\" (UID: \"f676c0de-5f1d-4ebc-aa86-6de59306b195\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-08b85c-5cc85974f5-ncbzk" Sep 10 13:42:28.725365 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:42:28.725338 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-08b85c-5cc85974f5-ncbzk"] Sep 10 13:42:28.811299 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:42:28.811267 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f676c0de-5f1d-4ebc-aa86-6de59306b195-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-08b85c-5cc85974f5-ncbzk\" (UID: \"f676c0de-5f1d-4ebc-aa86-6de59306b195\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-08b85c-5cc85974f5-ncbzk" Sep 10 13:42:28.811501 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:42:28.811386 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tqwvj\" (UniqueName: \"kubernetes.io/projected/f676c0de-5f1d-4ebc-aa86-6de59306b195-kube-api-access-tqwvj\") pod \"payload-processing-e2e-ipp-b-08b85c-5cc85974f5-ncbzk\" (UID: \"f676c0de-5f1d-4ebc-aa86-6de59306b195\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-08b85c-5cc85974f5-ncbzk" Sep 10 13:42:28.812026 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:42:28.811999 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f676c0de-5f1d-4ebc-aa86-6de59306b195-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-08b85c-5cc85974f5-ncbzk\" (UID: \"f676c0de-5f1d-4ebc-aa86-6de59306b195\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-08b85c-5cc85974f5-ncbzk" Sep 10 13:42:28.833095 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:42:28.833064 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tqwvj\" (UniqueName: \"kubernetes.io/projected/f676c0de-5f1d-4ebc-aa86-6de59306b195-kube-api-access-tqwvj\") pod \"payload-processing-e2e-ipp-b-08b85c-5cc85974f5-ncbzk\" (UID: \"f676c0de-5f1d-4ebc-aa86-6de59306b195\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-08b85c-5cc85974f5-ncbzk" Sep 10 13:42:28.984462 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:42:28.984407 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-08b85c-5cc85974f5-ncbzk" Sep 10 13:42:29.357525 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:42:29.357486 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-08b85c-5cc85974f5-ncbzk"] Sep 10 13:42:29.359859 ip-10-0-139-174 kubenswrapper[2557]: W0910 13:42:29.359828 2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf676c0de_5f1d_4ebc_aa86_6de59306b195.slice/crio-a0afbc0371f31eca5b506acfa5036ae4b5679d633f86b188d11ea6872f32e85d WatchSource:0}: Error finding container a0afbc0371f31eca5b506acfa5036ae4b5679d633f86b188d11ea6872f32e85d: Status 404 returned error can't find the container with id a0afbc0371f31eca5b506acfa5036ae4b5679d633f86b188d11ea6872f32e85d Sep 10 13:42:29.449981 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:42:29.449943 2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-e65037-7c58dd8dff-g4wd6"] Sep 10 13:42:29.450475 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:42:29.450197 2557 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-e65037-7c58dd8dff-g4wd6" podUID="1d6c6e09-c18e-4388-a708-1ad80cd647ed" containerName="payload-processing" containerID="cri-o://75110f0dcd52679d4559ab20d3f782320435abe3bf7d6ba734cb669c179835ee" gracePeriod=30 Sep 10 13:42:29.700027 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:42:29.700003 2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-e65037-7c58dd8dff-g4wd6" Sep 10 13:42:29.821499 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:42:29.821465 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-b8drm\" (UniqueName: \"kubernetes.io/projected/1d6c6e09-c18e-4388-a708-1ad80cd647ed-kube-api-access-b8drm\") pod \"1d6c6e09-c18e-4388-a708-1ad80cd647ed\" (UID: \"1d6c6e09-c18e-4388-a708-1ad80cd647ed\") " Sep 10 13:42:29.821706 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:42:29.821535 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1d6c6e09-c18e-4388-a708-1ad80cd647ed-plugins-config-volume\") pod \"1d6c6e09-c18e-4388-a708-1ad80cd647ed\" (UID: \"1d6c6e09-c18e-4388-a708-1ad80cd647ed\") " Sep 10 13:42:29.822187 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:42:29.822152 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1d6c6e09-c18e-4388-a708-1ad80cd647ed-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "1d6c6e09-c18e-4388-a708-1ad80cd647ed" (UID: "1d6c6e09-c18e-4388-a708-1ad80cd647ed"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 10 13:42:29.824066 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:42:29.824032 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1d6c6e09-c18e-4388-a708-1ad80cd647ed-kube-api-access-b8drm" (OuterVolumeSpecName: "kube-api-access-b8drm") pod "1d6c6e09-c18e-4388-a708-1ad80cd647ed" (UID: "1d6c6e09-c18e-4388-a708-1ad80cd647ed"). InnerVolumeSpecName "kube-api-access-b8drm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 10 13:42:29.922354 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:42:29.922289 2557 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b8drm\" (UniqueName: \"kubernetes.io/projected/1d6c6e09-c18e-4388-a708-1ad80cd647ed-kube-api-access-b8drm\") on node \"ip-10-0-139-174.ec2.internal\" DevicePath \"\"" Sep 10 13:42:29.922354 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:42:29.922309 2557 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1d6c6e09-c18e-4388-a708-1ad80cd647ed-plugins-config-volume\") on node \"ip-10-0-139-174.ec2.internal\" DevicePath \"\"" Sep 10 13:42:30.108731 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:42:30.108695 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-08b85c-5cc85974f5-ncbzk" event={"ID":"f676c0de-5f1d-4ebc-aa86-6de59306b195","Type":"ContainerStarted","Data":"0273e594d5ee7889342dbc48d39ddc4497b5296e182133538f490fdef44b3dbf"} Sep 10 13:42:30.108886 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:42:30.108752 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-08b85c-5cc85974f5-ncbzk" event={"ID":"f676c0de-5f1d-4ebc-aa86-6de59306b195","Type":"ContainerStarted","Data":"a0afbc0371f31eca5b506acfa5036ae4b5679d633f86b188d11ea6872f32e85d"} Sep 10 13:42:30.108886 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:42:30.108778 2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-08b85c-5cc85974f5-ncbzk" Sep 10 13:42:30.110102 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:42:30.110075 2557 generic.go:356] "Generic (PLEG): container finished" podID="1d6c6e09-c18e-4388-a708-1ad80cd647ed" containerID="75110f0dcd52679d4559ab20d3f782320435abe3bf7d6ba734cb669c179835ee" exitCode=0 Sep 10 13:42:30.110220 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:42:30.110113 2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-e65037-7c58dd8dff-g4wd6" Sep 10 13:42:30.110220 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:42:30.110155 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-e65037-7c58dd8dff-g4wd6" event={"ID":"1d6c6e09-c18e-4388-a708-1ad80cd647ed","Type":"ContainerDied","Data":"75110f0dcd52679d4559ab20d3f782320435abe3bf7d6ba734cb669c179835ee"} Sep 10 13:42:30.110220 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:42:30.110201 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-e65037-7c58dd8dff-g4wd6" event={"ID":"1d6c6e09-c18e-4388-a708-1ad80cd647ed","Type":"ContainerDied","Data":"873ef8831b7aac12d2eb0056ae796a29dd29db94bdd9ba6f56ee0227d30d76a4"} Sep 10 13:42:30.110334 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:42:30.110225 2557 scope.go:117] "RemoveContainer" containerID="75110f0dcd52679d4559ab20d3f782320435abe3bf7d6ba734cb669c179835ee" Sep 10 13:42:30.120341 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:42:30.120324 2557 scope.go:117] "RemoveContainer" containerID="75110f0dcd52679d4559ab20d3f782320435abe3bf7d6ba734cb669c179835ee" Sep 10 13:42:30.120644 ip-10-0-139-174 kubenswrapper[2557]: E0910 13:42:30.120628 2557 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"75110f0dcd52679d4559ab20d3f782320435abe3bf7d6ba734cb669c179835ee\": container with ID starting with 75110f0dcd52679d4559ab20d3f782320435abe3bf7d6ba734cb669c179835ee not found: ID does not exist" containerID="75110f0dcd52679d4559ab20d3f782320435abe3bf7d6ba734cb669c179835ee" Sep 10 13:42:30.120707 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:42:30.120651 2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"75110f0dcd52679d4559ab20d3f782320435abe3bf7d6ba734cb669c179835ee"} err="failed to get container status \"75110f0dcd52679d4559ab20d3f782320435abe3bf7d6ba734cb669c179835ee\": rpc error: code = NotFound desc = could not find container \"75110f0dcd52679d4559ab20d3f782320435abe3bf7d6ba734cb669c179835ee\": container with ID starting with 75110f0dcd52679d4559ab20d3f782320435abe3bf7d6ba734cb669c179835ee not found: ID does not exist" Sep 10 13:42:30.163155 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:42:30.163097 2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-b-08b85c-5cc85974f5-ncbzk" podStartSLOduration=2.163081163 podStartE2EDuration="2.163081163s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 13:42:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 13:42:30.159998248 +0000 UTC m=+1510.466992673" watchObservedRunningTime="2026-09-10 13:42:30.163081163 +0000 UTC m=+1510.470075625" Sep 10 13:42:30.229061 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:42:30.229029 2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-e65037-7c58dd8dff-g4wd6"] Sep 10 13:42:30.247591 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:42:30.247566 2557 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-e65037-7c58dd8dff-g4wd6"] Sep 10 13:42:32.220191 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:42:32.220160 2557 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1d6c6e09-c18e-4388-a708-1ad80cd647ed" path="/var/lib/kubelet/pods/1d6c6e09-c18e-4388-a708-1ad80cd647ed/volumes" Sep 10 13:42:37.997590 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:42:37.997557 2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-e65037-openshift-default-5c9697d44-nm2bl"] Sep 10 13:42:37.998085 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:42:37.997752 2557 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w2-e65037-openshift-default-5c9697d44-nm2bl" podUID="9c46315e-65c5-42f5-81de-91e639e6ed98" containerName="istio-proxy" containerID="cri-o://80764c4fc5dde5bd235d79f385ff53176e37622fb3633ef124968ab1ccbc9666" gracePeriod=30 Sep 10 13:42:38.670297 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:42:38.670261 2557 patch_prober.go:28] interesting pod/e2e-shared-b-w2-e65037-openshift-default-5c9697d44-nm2bl container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Sep 10 13:42:38.670481 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:42:38.670332 2557 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-b-w2-e65037-openshift-default-5c9697d44-nm2bl" podUID="9c46315e-65c5-42f5-81de-91e639e6ed98" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Sep 10 13:42:41.118859 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:42:41.118787 2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-08b85c-5cc85974f5-ncbzk" Sep 10 13:42:43.188522 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:42:43.188485 2557 generic.go:356] "Generic (PLEG): container finished" podID="9c46315e-65c5-42f5-81de-91e639e6ed98" containerID="80764c4fc5dde5bd235d79f385ff53176e37622fb3633ef124968ab1ccbc9666" exitCode=0 Sep 10 13:42:43.188941 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:42:43.188545 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-e65037-openshift-default-5c9697d44-nm2bl" event={"ID":"9c46315e-65c5-42f5-81de-91e639e6ed98","Type":"ContainerDied","Data":"80764c4fc5dde5bd235d79f385ff53176e37622fb3633ef124968ab1ccbc9666"} Sep 10 13:42:43.188941 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:42:43.188612 2557 scope.go:117] "RemoveContainer" containerID="e2175dab6789a058f8364fb79878fe74a180e2ad85a198d39c28f96d199e7415" Sep 10 13:42:43.251715 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:42:43.251694 2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-e65037-openshift-default-5c9697d44-nm2bl" Sep 10 13:42:43.338132 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:42:43.338057 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9c46315e-65c5-42f5-81de-91e639e6ed98-istiod-ca-cert\") pod \"9c46315e-65c5-42f5-81de-91e639e6ed98\" (UID: \"9c46315e-65c5-42f5-81de-91e639e6ed98\") " Sep 10 13:42:43.338132 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:42:43.338090 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9c46315e-65c5-42f5-81de-91e639e6ed98-workload-certs\") pod \"9c46315e-65c5-42f5-81de-91e639e6ed98\" (UID: \"9c46315e-65c5-42f5-81de-91e639e6ed98\") " Sep 10 13:42:43.338132 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:42:43.338120 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7r7d7\" (UniqueName: \"kubernetes.io/projected/9c46315e-65c5-42f5-81de-91e639e6ed98-kube-api-access-7r7d7\") pod \"9c46315e-65c5-42f5-81de-91e639e6ed98\" (UID: \"9c46315e-65c5-42f5-81de-91e639e6ed98\") " Sep 10 13:42:43.338409 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:42:43.338155 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9c46315e-65c5-42f5-81de-91e639e6ed98-istio-token\") pod \"9c46315e-65c5-42f5-81de-91e639e6ed98\" (UID: \"9c46315e-65c5-42f5-81de-91e639e6ed98\") " Sep 10 13:42:43.338409 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:42:43.338219 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9c46315e-65c5-42f5-81de-91e639e6ed98-workload-socket\") pod \"9c46315e-65c5-42f5-81de-91e639e6ed98\" (UID: \"9c46315e-65c5-42f5-81de-91e639e6ed98\") " Sep 10 13:42:43.338409 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:42:43.338247 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9c46315e-65c5-42f5-81de-91e639e6ed98-istio-envoy\") pod \"9c46315e-65c5-42f5-81de-91e639e6ed98\" (UID: \"9c46315e-65c5-42f5-81de-91e639e6ed98\") " Sep 10 13:42:43.338409 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:42:43.338310 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9c46315e-65c5-42f5-81de-91e639e6ed98-istio-data\") pod \"9c46315e-65c5-42f5-81de-91e639e6ed98\" (UID: \"9c46315e-65c5-42f5-81de-91e639e6ed98\") " Sep 10 13:42:43.338409 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:42:43.338331 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9c46315e-65c5-42f5-81de-91e639e6ed98-istio-podinfo\") pod \"9c46315e-65c5-42f5-81de-91e639e6ed98\" (UID: \"9c46315e-65c5-42f5-81de-91e639e6ed98\") " Sep 10 13:42:43.338409 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:42:43.338332 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9c46315e-65c5-42f5-81de-91e639e6ed98-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "9c46315e-65c5-42f5-81de-91e639e6ed98" (UID: "9c46315e-65c5-42f5-81de-91e639e6ed98"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 13:42:43.338409 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:42:43.338361 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9c46315e-65c5-42f5-81de-91e639e6ed98-credential-socket\") pod \"9c46315e-65c5-42f5-81de-91e639e6ed98\" (UID: \"9c46315e-65c5-42f5-81de-91e639e6ed98\") " Sep 10 13:42:43.338773 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:42:43.338561 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9c46315e-65c5-42f5-81de-91e639e6ed98-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "9c46315e-65c5-42f5-81de-91e639e6ed98" (UID: "9c46315e-65c5-42f5-81de-91e639e6ed98"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 13:42:43.338773 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:42:43.338597 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9c46315e-65c5-42f5-81de-91e639e6ed98-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "9c46315e-65c5-42f5-81de-91e639e6ed98" (UID: "9c46315e-65c5-42f5-81de-91e639e6ed98"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 10 13:42:43.338773 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:42:43.338729 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9c46315e-65c5-42f5-81de-91e639e6ed98-istio-data" (OuterVolumeSpecName: "istio-data") pod "9c46315e-65c5-42f5-81de-91e639e6ed98" (UID: "9c46315e-65c5-42f5-81de-91e639e6ed98"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 13:42:43.338773 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:42:43.338750 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9c46315e-65c5-42f5-81de-91e639e6ed98-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "9c46315e-65c5-42f5-81de-91e639e6ed98" (UID: "9c46315e-65c5-42f5-81de-91e639e6ed98"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 13:42:43.338985 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:42:43.338944 2557 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9c46315e-65c5-42f5-81de-91e639e6ed98-istiod-ca-cert\") on node \"ip-10-0-139-174.ec2.internal\" DevicePath \"\"" Sep 10 13:42:43.338985 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:42:43.338963 2557 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9c46315e-65c5-42f5-81de-91e639e6ed98-workload-certs\") on node \"ip-10-0-139-174.ec2.internal\" DevicePath \"\"" Sep 10 13:42:43.338985 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:42:43.338975 2557 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9c46315e-65c5-42f5-81de-91e639e6ed98-workload-socket\") on node \"ip-10-0-139-174.ec2.internal\" DevicePath \"\"" Sep 10 13:42:43.339151 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:42:43.338988 2557 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9c46315e-65c5-42f5-81de-91e639e6ed98-istio-data\") on node \"ip-10-0-139-174.ec2.internal\" DevicePath \"\"" Sep 10 13:42:43.339151 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:42:43.339001 2557 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9c46315e-65c5-42f5-81de-91e639e6ed98-credential-socket\") on node \"ip-10-0-139-174.ec2.internal\" DevicePath \"\"" Sep 10 13:42:43.340693 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:42:43.340666 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9c46315e-65c5-42f5-81de-91e639e6ed98-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "9c46315e-65c5-42f5-81de-91e639e6ed98" (UID: "9c46315e-65c5-42f5-81de-91e639e6ed98"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 13:42:43.340798 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:42:43.340697 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9c46315e-65c5-42f5-81de-91e639e6ed98-kube-api-access-7r7d7" (OuterVolumeSpecName: "kube-api-access-7r7d7") pod "9c46315e-65c5-42f5-81de-91e639e6ed98" (UID: "9c46315e-65c5-42f5-81de-91e639e6ed98"). InnerVolumeSpecName "kube-api-access-7r7d7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 10 13:42:43.340798 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:42:43.340750 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9c46315e-65c5-42f5-81de-91e639e6ed98-istio-token" (OuterVolumeSpecName: "istio-token") pod "9c46315e-65c5-42f5-81de-91e639e6ed98" (UID: "9c46315e-65c5-42f5-81de-91e639e6ed98"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 10 13:42:43.340798 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:42:43.340771 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/9c46315e-65c5-42f5-81de-91e639e6ed98-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "9c46315e-65c5-42f5-81de-91e639e6ed98" (UID: "9c46315e-65c5-42f5-81de-91e639e6ed98"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Sep 10 13:42:43.439466 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:42:43.439439 2557 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9c46315e-65c5-42f5-81de-91e639e6ed98-istio-podinfo\") on node \"ip-10-0-139-174.ec2.internal\" DevicePath \"\"" Sep 10 13:42:43.439466 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:42:43.439465 2557 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7r7d7\" (UniqueName: \"kubernetes.io/projected/9c46315e-65c5-42f5-81de-91e639e6ed98-kube-api-access-7r7d7\") on node \"ip-10-0-139-174.ec2.internal\" DevicePath \"\"" Sep 10 13:42:43.439622 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:42:43.439475 2557 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9c46315e-65c5-42f5-81de-91e639e6ed98-istio-token\") on node \"ip-10-0-139-174.ec2.internal\" DevicePath \"\"" Sep 10 13:42:43.439622 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:42:43.439483 2557 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9c46315e-65c5-42f5-81de-91e639e6ed98-istio-envoy\") on node \"ip-10-0-139-174.ec2.internal\" DevicePath \"\"" Sep 10 13:42:44.199190 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:42:44.199160 2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-e65037-openshift-default-5c9697d44-nm2bl" Sep 10 13:42:44.199662 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:42:44.199203 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-e65037-openshift-default-5c9697d44-nm2bl" event={"ID":"9c46315e-65c5-42f5-81de-91e639e6ed98","Type":"ContainerDied","Data":"4163482345bd5cdc6443e7e5acf67554dfb272a479cb14005088143778d958e4"} Sep 10 13:42:44.199662 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:42:44.199286 2557 scope.go:117] "RemoveContainer" containerID="80764c4fc5dde5bd235d79f385ff53176e37622fb3633ef124968ab1ccbc9666" Sep 10 13:42:44.221893 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:42:44.221873 2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-e65037-openshift-default-5c9697d44-nm2bl"] Sep 10 13:42:44.232892 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:42:44.232864 2557 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-e65037-openshift-default-5c9697d44-nm2bl"] Sep 10 13:42:46.220091 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:42:46.220062 2557 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9c46315e-65c5-42f5-81de-91e639e6ed98" path="/var/lib/kubelet/pods/9c46315e-65c5-42f5-81de-91e639e6ed98/volumes" Sep 10 13:42:56.661473 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:42:56.661442 2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5b54b7fbb7-t2dzv_68b5cdbd-ff82-4dc9-aa29-c592a12629e0/payload-processing/0.log" Sep 10 13:42:56.793552 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:42:56.793526 2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-e2e-ipp-b-08b85c-5cc85974f5-ncbzk_f676c0de-5f1d-4ebc-aa86-6de59306b195/payload-processing/0.log" Sep 10 13:43:32.732514 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:43:32.732480 2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5b54b7fbb7-t2dzv_68b5cdbd-ff82-4dc9-aa29-c592a12629e0/payload-processing/0.log" Sep 10 13:43:32.868153 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:43:32.868131 2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-e2e-ipp-b-08b85c-5cc85974f5-ncbzk_f676c0de-5f1d-4ebc-aa86-6de59306b195/payload-processing/0.log" Sep 10 13:43:33.563929 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:43:33.563899 2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-cleanup-37280b-openshift-default-769cf8bfd8-2xj6w"] Sep 10 13:43:33.564315 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:43:33.564299 2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1d6c6e09-c18e-4388-a708-1ad80cd647ed" containerName="payload-processing" Sep 10 13:43:33.564358 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:43:33.564320 2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="1d6c6e09-c18e-4388-a708-1ad80cd647ed" containerName="payload-processing" Sep 10 13:43:33.564358 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:43:33.564347 2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9c46315e-65c5-42f5-81de-91e639e6ed98" containerName="istio-proxy" Sep 10 13:43:33.564448 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:43:33.564358 2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="9c46315e-65c5-42f5-81de-91e639e6ed98" containerName="istio-proxy" Sep 10 13:43:33.564492 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:43:33.564472 2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="9c46315e-65c5-42f5-81de-91e639e6ed98" containerName="istio-proxy" Sep 10 13:43:33.564492 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:43:33.564485 2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="9c46315e-65c5-42f5-81de-91e639e6ed98" containerName="istio-proxy" Sep 10 13:43:33.564492 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:43:33.564491 2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="1d6c6e09-c18e-4388-a708-1ad80cd647ed" containerName="payload-processing" Sep 10 13:43:33.569463 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:43:33.569442 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-cleanup-37280b-openshift-default-769cf8bfd8-2xj6w" Sep 10 13:43:33.572248 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:43:33.572228 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-cleanup-37280b-openshift-default-dockercfg-qpzmq\"" Sep 10 13:43:33.589100 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:43:33.589077 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-cleanup-37280b-openshift-default-769cf8bfd8-2xj6w"] Sep 10 13:43:33.659598 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:43:33.659565 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-slsfj\" (UniqueName: \"kubernetes.io/projected/d904068a-a5e6-402e-bc7a-9e048423f51a-kube-api-access-slsfj\") pod \"e2e-ipp-cleanup-37280b-openshift-default-769cf8bfd8-2xj6w\" (UID: \"d904068a-a5e6-402e-bc7a-9e048423f51a\") " pod="openshift-ingress/e2e-ipp-cleanup-37280b-openshift-default-769cf8bfd8-2xj6w" Sep 10 13:43:33.659799 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:43:33.659645 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d904068a-a5e6-402e-bc7a-9e048423f51a-istiod-ca-cert\") pod \"e2e-ipp-cleanup-37280b-openshift-default-769cf8bfd8-2xj6w\" (UID: \"d904068a-a5e6-402e-bc7a-9e048423f51a\") " pod="openshift-ingress/e2e-ipp-cleanup-37280b-openshift-default-769cf8bfd8-2xj6w" Sep 10 13:43:33.659799 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:43:33.659695 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d904068a-a5e6-402e-bc7a-9e048423f51a-istio-podinfo\") pod \"e2e-ipp-cleanup-37280b-openshift-default-769cf8bfd8-2xj6w\" (UID: \"d904068a-a5e6-402e-bc7a-9e048423f51a\") " pod="openshift-ingress/e2e-ipp-cleanup-37280b-openshift-default-769cf8bfd8-2xj6w" Sep 10 13:43:33.659799 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:43:33.659750 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d904068a-a5e6-402e-bc7a-9e048423f51a-istio-data\") pod \"e2e-ipp-cleanup-37280b-openshift-default-769cf8bfd8-2xj6w\" (UID: \"d904068a-a5e6-402e-bc7a-9e048423f51a\") " pod="openshift-ingress/e2e-ipp-cleanup-37280b-openshift-default-769cf8bfd8-2xj6w" Sep 10 13:43:33.659978 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:43:33.659835 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d904068a-a5e6-402e-bc7a-9e048423f51a-workload-socket\") pod \"e2e-ipp-cleanup-37280b-openshift-default-769cf8bfd8-2xj6w\" (UID: \"d904068a-a5e6-402e-bc7a-9e048423f51a\") " pod="openshift-ingress/e2e-ipp-cleanup-37280b-openshift-default-769cf8bfd8-2xj6w" Sep 10 13:43:33.659978 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:43:33.659867 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d904068a-a5e6-402e-bc7a-9e048423f51a-workload-certs\") pod \"e2e-ipp-cleanup-37280b-openshift-default-769cf8bfd8-2xj6w\" (UID: \"d904068a-a5e6-402e-bc7a-9e048423f51a\") " pod="openshift-ingress/e2e-ipp-cleanup-37280b-openshift-default-769cf8bfd8-2xj6w" Sep 10 13:43:33.659978 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:43:33.659944 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d904068a-a5e6-402e-bc7a-9e048423f51a-istio-envoy\") pod \"e2e-ipp-cleanup-37280b-openshift-default-769cf8bfd8-2xj6w\" (UID: \"d904068a-a5e6-402e-bc7a-9e048423f51a\") " pod="openshift-ingress/e2e-ipp-cleanup-37280b-openshift-default-769cf8bfd8-2xj6w" Sep 10 13:43:33.660125 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:43:33.659986 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d904068a-a5e6-402e-bc7a-9e048423f51a-istio-token\") pod \"e2e-ipp-cleanup-37280b-openshift-default-769cf8bfd8-2xj6w\" (UID: \"d904068a-a5e6-402e-bc7a-9e048423f51a\") " pod="openshift-ingress/e2e-ipp-cleanup-37280b-openshift-default-769cf8bfd8-2xj6w" Sep 10 13:43:33.660125 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:43:33.660053 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d904068a-a5e6-402e-bc7a-9e048423f51a-credential-socket\") pod \"e2e-ipp-cleanup-37280b-openshift-default-769cf8bfd8-2xj6w\" (UID: \"d904068a-a5e6-402e-bc7a-9e048423f51a\") " pod="openshift-ingress/e2e-ipp-cleanup-37280b-openshift-default-769cf8bfd8-2xj6w" Sep 10 13:43:33.761465 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:43:33.761409 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d904068a-a5e6-402e-bc7a-9e048423f51a-workload-socket\") pod \"e2e-ipp-cleanup-37280b-openshift-default-769cf8bfd8-2xj6w\" (UID: \"d904068a-a5e6-402e-bc7a-9e048423f51a\") " pod="openshift-ingress/e2e-ipp-cleanup-37280b-openshift-default-769cf8bfd8-2xj6w" Sep 10 13:43:33.761935 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:43:33.761479 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d904068a-a5e6-402e-bc7a-9e048423f51a-workload-certs\") pod \"e2e-ipp-cleanup-37280b-openshift-default-769cf8bfd8-2xj6w\" (UID: \"d904068a-a5e6-402e-bc7a-9e048423f51a\") " pod="openshift-ingress/e2e-ipp-cleanup-37280b-openshift-default-769cf8bfd8-2xj6w" Sep 10 13:43:33.761935 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:43:33.761511 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d904068a-a5e6-402e-bc7a-9e048423f51a-istio-envoy\") pod \"e2e-ipp-cleanup-37280b-openshift-default-769cf8bfd8-2xj6w\" (UID: \"d904068a-a5e6-402e-bc7a-9e048423f51a\") " pod="openshift-ingress/e2e-ipp-cleanup-37280b-openshift-default-769cf8bfd8-2xj6w" Sep 10 13:43:33.761935 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:43:33.761533 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d904068a-a5e6-402e-bc7a-9e048423f51a-istio-token\") pod \"e2e-ipp-cleanup-37280b-openshift-default-769cf8bfd8-2xj6w\" (UID: \"d904068a-a5e6-402e-bc7a-9e048423f51a\") " pod="openshift-ingress/e2e-ipp-cleanup-37280b-openshift-default-769cf8bfd8-2xj6w" Sep 10 13:43:33.761935 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:43:33.761584 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d904068a-a5e6-402e-bc7a-9e048423f51a-credential-socket\") pod \"e2e-ipp-cleanup-37280b-openshift-default-769cf8bfd8-2xj6w\" (UID: \"d904068a-a5e6-402e-bc7a-9e048423f51a\") " pod="openshift-ingress/e2e-ipp-cleanup-37280b-openshift-default-769cf8bfd8-2xj6w" Sep 10 13:43:33.761935 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:43:33.761624 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-slsfj\" (UniqueName: \"kubernetes.io/projected/d904068a-a5e6-402e-bc7a-9e048423f51a-kube-api-access-slsfj\") pod \"e2e-ipp-cleanup-37280b-openshift-default-769cf8bfd8-2xj6w\" (UID: \"d904068a-a5e6-402e-bc7a-9e048423f51a\") " pod="openshift-ingress/e2e-ipp-cleanup-37280b-openshift-default-769cf8bfd8-2xj6w" Sep 10 13:43:33.761935 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:43:33.761697 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d904068a-a5e6-402e-bc7a-9e048423f51a-istiod-ca-cert\") pod \"e2e-ipp-cleanup-37280b-openshift-default-769cf8bfd8-2xj6w\" (UID: \"d904068a-a5e6-402e-bc7a-9e048423f51a\") " pod="openshift-ingress/e2e-ipp-cleanup-37280b-openshift-default-769cf8bfd8-2xj6w" Sep 10 13:43:33.761935 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:43:33.761738 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d904068a-a5e6-402e-bc7a-9e048423f51a-istio-podinfo\") pod \"e2e-ipp-cleanup-37280b-openshift-default-769cf8bfd8-2xj6w\" (UID: \"d904068a-a5e6-402e-bc7a-9e048423f51a\") " pod="openshift-ingress/e2e-ipp-cleanup-37280b-openshift-default-769cf8bfd8-2xj6w" Sep 10 13:43:33.761935 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:43:33.761776 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d904068a-a5e6-402e-bc7a-9e048423f51a-istio-data\") pod \"e2e-ipp-cleanup-37280b-openshift-default-769cf8bfd8-2xj6w\" (UID: \"d904068a-a5e6-402e-bc7a-9e048423f51a\") " pod="openshift-ingress/e2e-ipp-cleanup-37280b-openshift-default-769cf8bfd8-2xj6w" Sep 10 13:43:33.761935 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:43:33.761820 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d904068a-a5e6-402e-bc7a-9e048423f51a-workload-socket\") pod \"e2e-ipp-cleanup-37280b-openshift-default-769cf8bfd8-2xj6w\" (UID: \"d904068a-a5e6-402e-bc7a-9e048423f51a\") " pod="openshift-ingress/e2e-ipp-cleanup-37280b-openshift-default-769cf8bfd8-2xj6w" Sep 10 13:43:33.761935 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:43:33.761910 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d904068a-a5e6-402e-bc7a-9e048423f51a-workload-certs\") pod \"e2e-ipp-cleanup-37280b-openshift-default-769cf8bfd8-2xj6w\" (UID: \"d904068a-a5e6-402e-bc7a-9e048423f51a\") " pod="openshift-ingress/e2e-ipp-cleanup-37280b-openshift-default-769cf8bfd8-2xj6w" Sep 10 13:43:33.762343 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:43:33.762024 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d904068a-a5e6-402e-bc7a-9e048423f51a-credential-socket\") pod \"e2e-ipp-cleanup-37280b-openshift-default-769cf8bfd8-2xj6w\" (UID: \"d904068a-a5e6-402e-bc7a-9e048423f51a\") " pod="openshift-ingress/e2e-ipp-cleanup-37280b-openshift-default-769cf8bfd8-2xj6w" Sep 10 13:43:33.762343 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:43:33.762095 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d904068a-a5e6-402e-bc7a-9e048423f51a-istio-data\") pod \"e2e-ipp-cleanup-37280b-openshift-default-769cf8bfd8-2xj6w\" (UID: \"d904068a-a5e6-402e-bc7a-9e048423f51a\") " pod="openshift-ingress/e2e-ipp-cleanup-37280b-openshift-default-769cf8bfd8-2xj6w" Sep 10 13:43:33.762456 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:43:33.762390 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d904068a-a5e6-402e-bc7a-9e048423f51a-istiod-ca-cert\") pod \"e2e-ipp-cleanup-37280b-openshift-default-769cf8bfd8-2xj6w\" (UID: \"d904068a-a5e6-402e-bc7a-9e048423f51a\") " pod="openshift-ingress/e2e-ipp-cleanup-37280b-openshift-default-769cf8bfd8-2xj6w" Sep 10 13:43:33.763896 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:43:33.763874 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d904068a-a5e6-402e-bc7a-9e048423f51a-istio-envoy\") pod \"e2e-ipp-cleanup-37280b-openshift-default-769cf8bfd8-2xj6w\" (UID: \"d904068a-a5e6-402e-bc7a-9e048423f51a\") " pod="openshift-ingress/e2e-ipp-cleanup-37280b-openshift-default-769cf8bfd8-2xj6w" Sep 10 13:43:33.764555 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:43:33.764535 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d904068a-a5e6-402e-bc7a-9e048423f51a-istio-podinfo\") pod \"e2e-ipp-cleanup-37280b-openshift-default-769cf8bfd8-2xj6w\" (UID: \"d904068a-a5e6-402e-bc7a-9e048423f51a\") " pod="openshift-ingress/e2e-ipp-cleanup-37280b-openshift-default-769cf8bfd8-2xj6w" Sep 10 13:43:33.784001 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:43:33.783970 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d904068a-a5e6-402e-bc7a-9e048423f51a-istio-token\") pod \"e2e-ipp-cleanup-37280b-openshift-default-769cf8bfd8-2xj6w\" (UID: \"d904068a-a5e6-402e-bc7a-9e048423f51a\") " pod="openshift-ingress/e2e-ipp-cleanup-37280b-openshift-default-769cf8bfd8-2xj6w" Sep 10 13:43:33.784292 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:43:33.784275 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-slsfj\" (UniqueName: \"kubernetes.io/projected/d904068a-a5e6-402e-bc7a-9e048423f51a-kube-api-access-slsfj\") pod \"e2e-ipp-cleanup-37280b-openshift-default-769cf8bfd8-2xj6w\" (UID: \"d904068a-a5e6-402e-bc7a-9e048423f51a\") " pod="openshift-ingress/e2e-ipp-cleanup-37280b-openshift-default-769cf8bfd8-2xj6w" Sep 10 13:43:33.879940 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:43:33.879872 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-cleanup-37280b-openshift-default-769cf8bfd8-2xj6w" Sep 10 13:43:34.174385 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:43:34.174240 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-cleanup-37280b-openshift-default-769cf8bfd8-2xj6w"] Sep 10 13:43:34.176779 ip-10-0-139-174 kubenswrapper[2557]: W0910 13:43:34.176750 2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd904068a_a5e6_402e_bc7a_9e048423f51a.slice/crio-06021632d4ab1ba25b11010960e32be9483e47b42babd1f7dba844641ae8b5ab WatchSource:0}: Error finding container 06021632d4ab1ba25b11010960e32be9483e47b42babd1f7dba844641ae8b5ab: Status 404 returned error can't find the container with id 06021632d4ab1ba25b11010960e32be9483e47b42babd1f7dba844641ae8b5ab Sep 10 13:43:34.179024 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:43:34.178989 2557 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892152Ki","pods":"250"} Sep 10 13:43:34.179093 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:43:34.179057 2557 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892152Ki","pods":"250"} Sep 10 13:43:34.179093 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:43:34.179086 2557 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892152Ki","pods":"250"} Sep 10 13:43:34.477271 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:43:34.477243 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-cleanup-37280b-openshift-default-769cf8bfd8-2xj6w" event={"ID":"d904068a-a5e6-402e-bc7a-9e048423f51a","Type":"ContainerStarted","Data":"9fd58b379d987eb2dd4b7a9a3329da675117f8fe4ef36b1bfd7528965f27726b"} Sep 10 13:43:34.477471 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:43:34.477290 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-cleanup-37280b-openshift-default-769cf8bfd8-2xj6w" event={"ID":"d904068a-a5e6-402e-bc7a-9e048423f51a","Type":"ContainerStarted","Data":"06021632d4ab1ba25b11010960e32be9483e47b42babd1f7dba844641ae8b5ab"} Sep 10 13:43:34.528354 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:43:34.528310 2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-cleanup-37280b-openshift-default-769cf8bfd8-2xj6w" podStartSLOduration=1.528294574 podStartE2EDuration="1.528294574s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 13:43:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 13:43:34.526199238 +0000 UTC m=+1574.833193688" watchObservedRunningTime="2026-09-10 13:43:34.528294574 +0000 UTC m=+1574.835289034" Sep 10 13:43:34.880996 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:43:34.880916 2557 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-cleanup-37280b-openshift-default-769cf8bfd8-2xj6w" Sep 10 13:43:35.885124 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:43:35.885099 2557 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-cleanup-37280b-openshift-default-769cf8bfd8-2xj6w" Sep 10 13:43:36.489935 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:43:36.489909 2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-cleanup-37280b-openshift-default-769cf8bfd8-2xj6w" Sep 10 13:43:36.490809 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:43:36.490792 2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-cleanup-37280b-openshift-default-769cf8bfd8-2xj6w" Sep 10 13:43:38.433687 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:43:38.433659 2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-cleanup-37280b-7cd57ccddc-dtp8s"] Sep 10 13:43:38.434067 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:43:38.434054 2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9c46315e-65c5-42f5-81de-91e639e6ed98" containerName="istio-proxy" Sep 10 13:43:38.434067 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:43:38.434069 2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="9c46315e-65c5-42f5-81de-91e639e6ed98" containerName="istio-proxy" Sep 10 13:43:38.437240 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:43:38.437216 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-37280b-7cd57ccddc-dtp8s" Sep 10 13:43:38.439789 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:43:38.439764 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-cleanup-37280b-dockercfg-wn949\"" Sep 10 13:43:38.439919 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:43:38.439819 2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-cleanup-37280b\"" Sep 10 13:43:38.449970 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:43:38.449945 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-cleanup-37280b-7cd57ccddc-dtp8s"] Sep 10 13:43:38.606256 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:43:38.606213 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3c0a10ca-ab18-4fa2-844d-3d9cbabb871a-plugins-config-volume\") pod \"payload-processing-e2e-ipp-cleanup-37280b-7cd57ccddc-dtp8s\" (UID: \"3c0a10ca-ab18-4fa2-844d-3d9cbabb871a\") " pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-37280b-7cd57ccddc-dtp8s" Sep 10 13:43:38.606534 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:43:38.606501 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k7lrz\" (UniqueName: \"kubernetes.io/projected/3c0a10ca-ab18-4fa2-844d-3d9cbabb871a-kube-api-access-k7lrz\") pod \"payload-processing-e2e-ipp-cleanup-37280b-7cd57ccddc-dtp8s\" (UID: \"3c0a10ca-ab18-4fa2-844d-3d9cbabb871a\") " pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-37280b-7cd57ccddc-dtp8s" Sep 10 13:43:38.707333 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:43:38.707300 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3c0a10ca-ab18-4fa2-844d-3d9cbabb871a-plugins-config-volume\") pod \"payload-processing-e2e-ipp-cleanup-37280b-7cd57ccddc-dtp8s\" (UID: \"3c0a10ca-ab18-4fa2-844d-3d9cbabb871a\") " pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-37280b-7cd57ccddc-dtp8s" Sep 10 13:43:38.707551 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:43:38.707351 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k7lrz\" (UniqueName: \"kubernetes.io/projected/3c0a10ca-ab18-4fa2-844d-3d9cbabb871a-kube-api-access-k7lrz\") pod \"payload-processing-e2e-ipp-cleanup-37280b-7cd57ccddc-dtp8s\" (UID: \"3c0a10ca-ab18-4fa2-844d-3d9cbabb871a\") " pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-37280b-7cd57ccddc-dtp8s" Sep 10 13:43:38.707960 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:43:38.707938 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3c0a10ca-ab18-4fa2-844d-3d9cbabb871a-plugins-config-volume\") pod \"payload-processing-e2e-ipp-cleanup-37280b-7cd57ccddc-dtp8s\" (UID: \"3c0a10ca-ab18-4fa2-844d-3d9cbabb871a\") " pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-37280b-7cd57ccddc-dtp8s" Sep 10 13:43:38.737474 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:43:38.737443 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-k7lrz\" (UniqueName: \"kubernetes.io/projected/3c0a10ca-ab18-4fa2-844d-3d9cbabb871a-kube-api-access-k7lrz\") pod \"payload-processing-e2e-ipp-cleanup-37280b-7cd57ccddc-dtp8s\" (UID: \"3c0a10ca-ab18-4fa2-844d-3d9cbabb871a\") " pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-37280b-7cd57ccddc-dtp8s" Sep 10 13:43:38.750435 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:43:38.750394 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-37280b-7cd57ccddc-dtp8s" Sep 10 13:43:39.028842 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:43:39.028547 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-cleanup-37280b-7cd57ccddc-dtp8s"] Sep 10 13:43:39.030466 ip-10-0-139-174 kubenswrapper[2557]: W0910 13:43:39.030436 2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3c0a10ca_ab18_4fa2_844d_3d9cbabb871a.slice/crio-6112f53a34d1593156c841d5ebf10c174bf8fb7066ed452e15a22b661b9f1889 WatchSource:0}: Error finding container 6112f53a34d1593156c841d5ebf10c174bf8fb7066ed452e15a22b661b9f1889: Status 404 returned error can't find the container with id 6112f53a34d1593156c841d5ebf10c174bf8fb7066ed452e15a22b661b9f1889 Sep 10 13:43:39.514400 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:43:39.514364 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-37280b-7cd57ccddc-dtp8s" event={"ID":"3c0a10ca-ab18-4fa2-844d-3d9cbabb871a","Type":"ContainerStarted","Data":"1665134c2d5bda75cda710d4395c9aab196bb359b4823162c70d29b271c7fa72"} Sep 10 13:43:39.514781 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:43:39.514451 2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-37280b-7cd57ccddc-dtp8s" Sep 10 13:43:39.514781 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:43:39.514478 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-37280b-7cd57ccddc-dtp8s" event={"ID":"3c0a10ca-ab18-4fa2-844d-3d9cbabb871a","Type":"ContainerStarted","Data":"6112f53a34d1593156c841d5ebf10c174bf8fb7066ed452e15a22b661b9f1889"} Sep 10 13:43:39.536825 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:43:39.536779 2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-37280b-7cd57ccddc-dtp8s" podStartSLOduration=1.536765004 podStartE2EDuration="1.536765004s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 13:43:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 13:43:39.534507638 +0000 UTC m=+1579.841502087" watchObservedRunningTime="2026-09-10 13:43:39.536765004 +0000 UTC m=+1579.843759465" Sep 10 13:43:50.523173 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:43:50.523145 2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-37280b-7cd57ccddc-dtp8s" Sep 10 13:43:56.623951 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:43:56.623921 2557 generic.go:356] "Generic (PLEG): container finished" podID="d904068a-a5e6-402e-bc7a-9e048423f51a" containerID="9fd58b379d987eb2dd4b7a9a3329da675117f8fe4ef36b1bfd7528965f27726b" exitCode=0 Sep 10 13:43:56.624507 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:43:56.623994 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-cleanup-37280b-openshift-default-769cf8bfd8-2xj6w" event={"ID":"d904068a-a5e6-402e-bc7a-9e048423f51a","Type":"ContainerDied","Data":"9fd58b379d987eb2dd4b7a9a3329da675117f8fe4ef36b1bfd7528965f27726b"} Sep 10 13:43:56.624588 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:43:56.624523 2557 scope.go:117] "RemoveContainer" containerID="9fd58b379d987eb2dd4b7a9a3329da675117f8fe4ef36b1bfd7528965f27726b" Sep 10 13:43:56.625627 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:43:56.625594 2557 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892152Ki","pods":"250"} Sep 10 13:43:56.625724 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:43:56.625660 2557 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892152Ki","pods":"250"} Sep 10 13:43:56.625724 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:43:56.625695 2557 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892152Ki","pods":"250"} Sep 10 13:43:57.631650 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:43:57.631614 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-cleanup-37280b-openshift-default-769cf8bfd8-2xj6w" event={"ID":"d904068a-a5e6-402e-bc7a-9e048423f51a","Type":"ContainerStarted","Data":"ac0bdcb66dc8bd3e36c25453383751ce99223f3b0876e537e4e66d2f0e2efbf0"} Sep 10 13:43:57.880371 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:43:57.880331 2557 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-cleanup-37280b-openshift-default-769cf8bfd8-2xj6w" Sep 10 13:43:57.880584 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:43:57.880552 2557 patch_prober.go:28] interesting pod/e2e-ipp-cleanup-37280b-openshift-default-769cf8bfd8-2xj6w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.78:15021/healthz/ready\": dial tcp 10.134.0.78:15021: connect: connection refused" start-of-body= Sep 10 13:43:57.880637 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:43:57.880607 2557 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-cleanup-37280b-openshift-default-769cf8bfd8-2xj6w" podUID="d904068a-a5e6-402e-bc7a-9e048423f51a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.78:15021/healthz/ready\": dial tcp 10.134.0.78:15021: connect: connection refused" Sep 10 13:43:58.881164 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:43:58.881129 2557 patch_prober.go:28] interesting pod/e2e-ipp-cleanup-37280b-openshift-default-769cf8bfd8-2xj6w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.78:15021/healthz/ready\": dial tcp 10.134.0.78:15021: connect: connection refused" start-of-body= Sep 10 13:43:58.881586 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:43:58.881195 2557 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-cleanup-37280b-openshift-default-769cf8bfd8-2xj6w" podUID="d904068a-a5e6-402e-bc7a-9e048423f51a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.78:15021/healthz/ready\": dial tcp 10.134.0.78:15021: connect: connection refused" Sep 10 13:43:59.881024 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:43:59.880982 2557 patch_prober.go:28] interesting pod/e2e-ipp-cleanup-37280b-openshift-default-769cf8bfd8-2xj6w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.78:15021/healthz/ready\": dial tcp 10.134.0.78:15021: connect: connection refused" start-of-body= Sep 10 13:43:59.881237 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:43:59.881060 2557 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-cleanup-37280b-openshift-default-769cf8bfd8-2xj6w" podUID="d904068a-a5e6-402e-bc7a-9e048423f51a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.78:15021/healthz/ready\": dial tcp 10.134.0.78:15021: connect: connection refused" Sep 10 13:44:00.881012 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:44:00.880977 2557 patch_prober.go:28] interesting pod/e2e-ipp-cleanup-37280b-openshift-default-769cf8bfd8-2xj6w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.78:15021/healthz/ready\": dial tcp 10.134.0.78:15021: connect: connection refused" start-of-body= Sep 10 13:44:00.881213 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:44:00.881044 2557 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-cleanup-37280b-openshift-default-769cf8bfd8-2xj6w" podUID="d904068a-a5e6-402e-bc7a-9e048423f51a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.78:15021/healthz/ready\": dial tcp 10.134.0.78:15021: connect: connection refused" Sep 10 13:44:01.880638 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:44:01.880603 2557 patch_prober.go:28] interesting pod/e2e-ipp-cleanup-37280b-openshift-default-769cf8bfd8-2xj6w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.78:15021/healthz/ready\": dial tcp 10.134.0.78:15021: connect: connection refused" start-of-body= Sep 10 13:44:01.881018 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:44:01.880665 2557 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-cleanup-37280b-openshift-default-769cf8bfd8-2xj6w" podUID="d904068a-a5e6-402e-bc7a-9e048423f51a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.78:15021/healthz/ready\": dial tcp 10.134.0.78:15021: connect: connection refused" Sep 10 13:44:02.384534 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:44:02.384496 2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-37280b-nm6pr"] Sep 10 13:44:02.388051 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:44:02.388031 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-37280b-nm6pr" Sep 10 13:44:02.390330 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:44:02.390308 2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Sep 10 13:44:02.390619 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:44:02.390584 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-wtblv\"" Sep 10 13:44:02.403021 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:44:02.403000 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-37280b-nm6pr"] Sep 10 13:44:02.423351 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:44:02.423329 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b067d3bd-9dc0-410d-9024-bc1a2c6d0193-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-cleanup-37280b-nm6pr\" (UID: \"b067d3bd-9dc0-410d-9024-bc1a2c6d0193\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-37280b-nm6pr" Sep 10 13:44:02.524815 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:44:02.524780 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b067d3bd-9dc0-410d-9024-bc1a2c6d0193-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-cleanup-37280b-nm6pr\" (UID: \"b067d3bd-9dc0-410d-9024-bc1a2c6d0193\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-37280b-nm6pr" Sep 10 13:44:02.525491 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:44:02.525464 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b067d3bd-9dc0-410d-9024-bc1a2c6d0193-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-cleanup-37280b-nm6pr\" (UID: \"b067d3bd-9dc0-410d-9024-bc1a2c6d0193\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-37280b-nm6pr" Sep 10 13:44:02.699364 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:44:02.699334 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-37280b-nm6pr" Sep 10 13:44:02.884989 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:44:02.884961 2557 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-cleanup-37280b-openshift-default-769cf8bfd8-2xj6w" Sep 10 13:44:02.885370 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:44:02.885250 2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-cleanup-37280b-openshift-default-769cf8bfd8-2xj6w" Sep 10 13:44:02.885883 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:44:02.885865 2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-cleanup-37280b-openshift-default-769cf8bfd8-2xj6w" Sep 10 13:44:03.010596 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:44:03.010555 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-37280b-nm6pr"] Sep 10 13:44:03.012371 ip-10-0-139-174 kubenswrapper[2557]: W0910 13:44:03.012336 2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb067d3bd_9dc0_410d_9024_bc1a2c6d0193.slice/crio-bfd720ae4dff5b735bd9817e78f3b3c4a58f90cceaa8241d0259cefe68d355ef WatchSource:0}: Error finding container bfd720ae4dff5b735bd9817e78f3b3c4a58f90cceaa8241d0259cefe68d355ef: Status 404 returned error can't find the container with id bfd720ae4dff5b735bd9817e78f3b3c4a58f90cceaa8241d0259cefe68d355ef Sep 10 13:44:03.691032 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:44:03.690999 2557 generic.go:356] "Generic (PLEG): container finished" podID="b067d3bd-9dc0-410d-9024-bc1a2c6d0193" containerID="3e0725ebff17c25c3d5e40e06dbda967690e178efc401e957740dc3754e81dc5" exitCode=0 Sep 10 13:44:03.691179 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:44:03.691083 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-37280b-nm6pr" event={"ID":"b067d3bd-9dc0-410d-9024-bc1a2c6d0193","Type":"ContainerDied","Data":"3e0725ebff17c25c3d5e40e06dbda967690e178efc401e957740dc3754e81dc5"} Sep 10 13:44:03.691179 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:44:03.691149 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-37280b-nm6pr" event={"ID":"b067d3bd-9dc0-410d-9024-bc1a2c6d0193","Type":"ContainerStarted","Data":"bfd720ae4dff5b735bd9817e78f3b3c4a58f90cceaa8241d0259cefe68d355ef"} Sep 10 13:44:04.834212 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:44:04.834189 2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-37280b-nm6pr" Sep 10 13:44:04.948743 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:44:04.948676 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b067d3bd-9dc0-410d-9024-bc1a2c6d0193-maas-api-service-ca\") pod \"b067d3bd-9dc0-410d-9024-bc1a2c6d0193\" (UID: \"b067d3bd-9dc0-410d-9024-bc1a2c6d0193\") " Sep 10 13:44:04.949003 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:44:04.948981 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b067d3bd-9dc0-410d-9024-bc1a2c6d0193-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "b067d3bd-9dc0-410d-9024-bc1a2c6d0193" (UID: "b067d3bd-9dc0-410d-9024-bc1a2c6d0193"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 10 13:44:05.049742 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:44:05.049718 2557 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b067d3bd-9dc0-410d-9024-bc1a2c6d0193-maas-api-service-ca\") on node \"ip-10-0-139-174.ec2.internal\" DevicePath \"\"" Sep 10 13:44:05.704333 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:44:05.704303 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-37280b-nm6pr" event={"ID":"b067d3bd-9dc0-410d-9024-bc1a2c6d0193","Type":"ContainerDied","Data":"bfd720ae4dff5b735bd9817e78f3b3c4a58f90cceaa8241d0259cefe68d355ef"} Sep 10 13:44:05.704333 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:44:05.704330 2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-37280b-nm6pr" Sep 10 13:44:05.704539 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:44:05.704347 2557 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="bfd720ae4dff5b735bd9817e78f3b3c4a58f90cceaa8241d0259cefe68d355ef" Sep 10 13:44:08.707312 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:44:08.707276 2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-cleanup-37280b-7cd57ccddc-dtp8s"] Sep 10 13:44:08.707791 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:44:08.707565 2557 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-37280b-7cd57ccddc-dtp8s" podUID="3c0a10ca-ab18-4fa2-844d-3d9cbabb871a" containerName="payload-processing" containerID="cri-o://1665134c2d5bda75cda710d4395c9aab196bb359b4823162c70d29b271c7fa72" gracePeriod=30 Sep 10 13:44:08.986769 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:44:08.986746 2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-37280b-7cd57ccddc-dtp8s" Sep 10 13:44:09.085476 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:44:09.085449 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-k7lrz\" (UniqueName: \"kubernetes.io/projected/3c0a10ca-ab18-4fa2-844d-3d9cbabb871a-kube-api-access-k7lrz\") pod \"3c0a10ca-ab18-4fa2-844d-3d9cbabb871a\" (UID: \"3c0a10ca-ab18-4fa2-844d-3d9cbabb871a\") " Sep 10 13:44:09.085616 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:44:09.085535 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3c0a10ca-ab18-4fa2-844d-3d9cbabb871a-plugins-config-volume\") pod \"3c0a10ca-ab18-4fa2-844d-3d9cbabb871a\" (UID: \"3c0a10ca-ab18-4fa2-844d-3d9cbabb871a\") " Sep 10 13:44:09.085917 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:44:09.085887 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3c0a10ca-ab18-4fa2-844d-3d9cbabb871a-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "3c0a10ca-ab18-4fa2-844d-3d9cbabb871a" (UID: "3c0a10ca-ab18-4fa2-844d-3d9cbabb871a"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 10 13:44:09.087623 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:44:09.087603 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3c0a10ca-ab18-4fa2-844d-3d9cbabb871a-kube-api-access-k7lrz" (OuterVolumeSpecName: "kube-api-access-k7lrz") pod "3c0a10ca-ab18-4fa2-844d-3d9cbabb871a" (UID: "3c0a10ca-ab18-4fa2-844d-3d9cbabb871a"). InnerVolumeSpecName "kube-api-access-k7lrz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 10 13:44:09.098468 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:44:09.098448 2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-37280b-nm6pr"] Sep 10 13:44:09.108261 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:44:09.108229 2557 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-37280b-nm6pr"] Sep 10 13:44:09.186560 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:44:09.186527 2557 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k7lrz\" (UniqueName: \"kubernetes.io/projected/3c0a10ca-ab18-4fa2-844d-3d9cbabb871a-kube-api-access-k7lrz\") on node \"ip-10-0-139-174.ec2.internal\" DevicePath \"\"" Sep 10 13:44:09.186560 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:44:09.186552 2557 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3c0a10ca-ab18-4fa2-844d-3d9cbabb871a-plugins-config-volume\") on node \"ip-10-0-139-174.ec2.internal\" DevicePath \"\"" Sep 10 13:44:09.731123 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:44:09.731085 2557 generic.go:356] "Generic (PLEG): container finished" podID="3c0a10ca-ab18-4fa2-844d-3d9cbabb871a" containerID="1665134c2d5bda75cda710d4395c9aab196bb359b4823162c70d29b271c7fa72" exitCode=0 Sep 10 13:44:09.731525 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:44:09.731128 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-37280b-7cd57ccddc-dtp8s" event={"ID":"3c0a10ca-ab18-4fa2-844d-3d9cbabb871a","Type":"ContainerDied","Data":"1665134c2d5bda75cda710d4395c9aab196bb359b4823162c70d29b271c7fa72"} Sep 10 13:44:09.731525 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:44:09.731186 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-37280b-7cd57ccddc-dtp8s" event={"ID":"3c0a10ca-ab18-4fa2-844d-3d9cbabb871a","Type":"ContainerDied","Data":"6112f53a34d1593156c841d5ebf10c174bf8fb7066ed452e15a22b661b9f1889"} Sep 10 13:44:09.731525 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:44:09.731209 2557 scope.go:117] "RemoveContainer" containerID="1665134c2d5bda75cda710d4395c9aab196bb359b4823162c70d29b271c7fa72" Sep 10 13:44:09.731525 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:44:09.731136 2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-37280b-7cd57ccddc-dtp8s" Sep 10 13:44:09.741223 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:44:09.741207 2557 scope.go:117] "RemoveContainer" containerID="1665134c2d5bda75cda710d4395c9aab196bb359b4823162c70d29b271c7fa72" Sep 10 13:44:09.741480 ip-10-0-139-174 kubenswrapper[2557]: E0910 13:44:09.741461 2557 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1665134c2d5bda75cda710d4395c9aab196bb359b4823162c70d29b271c7fa72\": container with ID starting with 1665134c2d5bda75cda710d4395c9aab196bb359b4823162c70d29b271c7fa72 not found: ID does not exist" containerID="1665134c2d5bda75cda710d4395c9aab196bb359b4823162c70d29b271c7fa72" Sep 10 13:44:09.741553 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:44:09.741492 2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1665134c2d5bda75cda710d4395c9aab196bb359b4823162c70d29b271c7fa72"} err="failed to get container status \"1665134c2d5bda75cda710d4395c9aab196bb359b4823162c70d29b271c7fa72\": rpc error: code = NotFound desc = could not find container \"1665134c2d5bda75cda710d4395c9aab196bb359b4823162c70d29b271c7fa72\": container with ID starting with 1665134c2d5bda75cda710d4395c9aab196bb359b4823162c70d29b271c7fa72 not found: ID does not exist" Sep 10 13:44:09.762195 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:44:09.762173 2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-cleanup-37280b-7cd57ccddc-dtp8s"] Sep 10 13:44:09.767915 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:44:09.767894 2557 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-cleanup-37280b-7cd57ccddc-dtp8s"] Sep 10 13:44:10.220368 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:44:10.220330 2557 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3c0a10ca-ab18-4fa2-844d-3d9cbabb871a" path="/var/lib/kubelet/pods/3c0a10ca-ab18-4fa2-844d-3d9cbabb871a/volumes" Sep 10 13:44:10.220784 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:44:10.220765 2557 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b067d3bd-9dc0-410d-9024-bc1a2c6d0193" path="/var/lib/kubelet/pods/b067d3bd-9dc0-410d-9024-bc1a2c6d0193/volumes" Sep 10 13:44:16.374587 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:44:16.374501 2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-cleanup-37280b-openshift-default-769cf8bfd8-2xj6w"] Sep 10 13:44:16.374992 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:44:16.374757 2557 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-cleanup-37280b-openshift-default-769cf8bfd8-2xj6w" podUID="d904068a-a5e6-402e-bc7a-9e048423f51a" containerName="istio-proxy" containerID="cri-o://ac0bdcb66dc8bd3e36c25453383751ce99223f3b0876e537e4e66d2f0e2efbf0" gracePeriod=30 Sep 10 13:44:17.887755 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:44:17.887721 2557 patch_prober.go:28] interesting pod/e2e-ipp-cleanup-37280b-openshift-default-769cf8bfd8-2xj6w container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Sep 10 13:44:17.888123 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:44:17.887783 2557 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ipp-cleanup-37280b-openshift-default-769cf8bfd8-2xj6w" podUID="d904068a-a5e6-402e-bc7a-9e048423f51a" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Sep 10 13:44:20.442379 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:44:20.442349 2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-56b405-wm429"] Sep 10 13:44:20.442760 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:44:20.442747 2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b067d3bd-9dc0-410d-9024-bc1a2c6d0193" containerName="revoke-keys" Sep 10 13:44:20.442760 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:44:20.442761 2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="b067d3bd-9dc0-410d-9024-bc1a2c6d0193" containerName="revoke-keys" Sep 10 13:44:20.442833 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:44:20.442774 2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3c0a10ca-ab18-4fa2-844d-3d9cbabb871a" containerName="payload-processing" Sep 10 13:44:20.442833 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:44:20.442779 2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="3c0a10ca-ab18-4fa2-844d-3d9cbabb871a" containerName="payload-processing" Sep 10 13:44:20.442938 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:44:20.442849 2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="b067d3bd-9dc0-410d-9024-bc1a2c6d0193" containerName="revoke-keys" Sep 10 13:44:20.442938 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:44:20.442859 2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="3c0a10ca-ab18-4fa2-844d-3d9cbabb871a" containerName="payload-processing" Sep 10 13:44:20.445855 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:44:20.445840 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-56b405-wm429" Sep 10 13:44:20.448619 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:44:20.448595 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-wtblv\"" Sep 10 13:44:20.448741 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:44:20.448669 2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Sep 10 13:44:20.458533 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:44:20.458508 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-56b405-wm429"] Sep 10 13:44:20.592252 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:44:20.592226 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/78c88d9c-e196-4446-99eb-7d15f395077a-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-56b405-wm429\" (UID: \"78c88d9c-e196-4446-99eb-7d15f395077a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-56b405-wm429" Sep 10 13:44:20.693666 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:44:20.693598 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/78c88d9c-e196-4446-99eb-7d15f395077a-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-56b405-wm429\" (UID: \"78c88d9c-e196-4446-99eb-7d15f395077a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-56b405-wm429" Sep 10 13:44:20.694153 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:44:20.694134 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/78c88d9c-e196-4446-99eb-7d15f395077a-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-56b405-wm429\" (UID: \"78c88d9c-e196-4446-99eb-7d15f395077a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-56b405-wm429" Sep 10 13:44:20.757747 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:44:20.757716 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-56b405-wm429" Sep 10 13:44:21.036777 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:44:21.036748 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-56b405-wm429"] Sep 10 13:44:21.038409 ip-10-0-139-174 kubenswrapper[2557]: W0910 13:44:21.038384 2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod78c88d9c_e196_4446_99eb_7d15f395077a.slice/crio-828627d290e20cd6054eab58be606edb97688c31b9cd5d6ac1e52bdf2292cb76 WatchSource:0}: Error finding container 828627d290e20cd6054eab58be606edb97688c31b9cd5d6ac1e52bdf2292cb76: Status 404 returned error can't find the container with id 828627d290e20cd6054eab58be606edb97688c31b9cd5d6ac1e52bdf2292cb76 Sep 10 13:44:21.606847 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:44:21.606827 2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-cleanup-37280b-openshift-default-769cf8bfd8-2xj6w" Sep 10 13:44:21.703201 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:44:21.703176 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d904068a-a5e6-402e-bc7a-9e048423f51a-istio-envoy\") pod \"d904068a-a5e6-402e-bc7a-9e048423f51a\" (UID: \"d904068a-a5e6-402e-bc7a-9e048423f51a\") " Sep 10 13:44:21.703379 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:44:21.703218 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d904068a-a5e6-402e-bc7a-9e048423f51a-workload-socket\") pod \"d904068a-a5e6-402e-bc7a-9e048423f51a\" (UID: \"d904068a-a5e6-402e-bc7a-9e048423f51a\") " Sep 10 13:44:21.703379 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:44:21.703246 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-slsfj\" (UniqueName: \"kubernetes.io/projected/d904068a-a5e6-402e-bc7a-9e048423f51a-kube-api-access-slsfj\") pod \"d904068a-a5e6-402e-bc7a-9e048423f51a\" (UID: \"d904068a-a5e6-402e-bc7a-9e048423f51a\") " Sep 10 13:44:21.703379 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:44:21.703281 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d904068a-a5e6-402e-bc7a-9e048423f51a-istio-data\") pod \"d904068a-a5e6-402e-bc7a-9e048423f51a\" (UID: \"d904068a-a5e6-402e-bc7a-9e048423f51a\") " Sep 10 13:44:21.703379 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:44:21.703300 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d904068a-a5e6-402e-bc7a-9e048423f51a-istio-podinfo\") pod \"d904068a-a5e6-402e-bc7a-9e048423f51a\" (UID: \"d904068a-a5e6-402e-bc7a-9e048423f51a\") " Sep 10 13:44:21.703379 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:44:21.703323 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d904068a-a5e6-402e-bc7a-9e048423f51a-istiod-ca-cert\") pod \"d904068a-a5e6-402e-bc7a-9e048423f51a\" (UID: \"d904068a-a5e6-402e-bc7a-9e048423f51a\") " Sep 10 13:44:21.703692 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:44:21.703381 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d904068a-a5e6-402e-bc7a-9e048423f51a-workload-certs\") pod \"d904068a-a5e6-402e-bc7a-9e048423f51a\" (UID: \"d904068a-a5e6-402e-bc7a-9e048423f51a\") " Sep 10 13:44:21.703692 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:44:21.703502 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d904068a-a5e6-402e-bc7a-9e048423f51a-istio-token\") pod \"d904068a-a5e6-402e-bc7a-9e048423f51a\" (UID: \"d904068a-a5e6-402e-bc7a-9e048423f51a\") " Sep 10 13:44:21.703692 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:44:21.703512 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d904068a-a5e6-402e-bc7a-9e048423f51a-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "d904068a-a5e6-402e-bc7a-9e048423f51a" (UID: "d904068a-a5e6-402e-bc7a-9e048423f51a"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 13:44:21.703692 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:44:21.703580 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d904068a-a5e6-402e-bc7a-9e048423f51a-istio-data" (OuterVolumeSpecName: "istio-data") pod "d904068a-a5e6-402e-bc7a-9e048423f51a" (UID: "d904068a-a5e6-402e-bc7a-9e048423f51a"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 13:44:21.703692 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:44:21.703618 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d904068a-a5e6-402e-bc7a-9e048423f51a-credential-socket\") pod \"d904068a-a5e6-402e-bc7a-9e048423f51a\" (UID: \"d904068a-a5e6-402e-bc7a-9e048423f51a\") " Sep 10 13:44:21.703959 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:44:21.703789 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d904068a-a5e6-402e-bc7a-9e048423f51a-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "d904068a-a5e6-402e-bc7a-9e048423f51a" (UID: "d904068a-a5e6-402e-bc7a-9e048423f51a"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 13:44:21.703959 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:44:21.703869 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d904068a-a5e6-402e-bc7a-9e048423f51a-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "d904068a-a5e6-402e-bc7a-9e048423f51a" (UID: "d904068a-a5e6-402e-bc7a-9e048423f51a"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 10 13:44:21.704064 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:44:21.703983 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d904068a-a5e6-402e-bc7a-9e048423f51a-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "d904068a-a5e6-402e-bc7a-9e048423f51a" (UID: "d904068a-a5e6-402e-bc7a-9e048423f51a"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 13:44:21.704308 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:44:21.704197 2557 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d904068a-a5e6-402e-bc7a-9e048423f51a-workload-socket\") on node \"ip-10-0-139-174.ec2.internal\" DevicePath \"\"" Sep 10 13:44:21.704308 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:44:21.704225 2557 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d904068a-a5e6-402e-bc7a-9e048423f51a-istio-data\") on node \"ip-10-0-139-174.ec2.internal\" DevicePath \"\"" Sep 10 13:44:21.704308 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:44:21.704239 2557 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d904068a-a5e6-402e-bc7a-9e048423f51a-istiod-ca-cert\") on node \"ip-10-0-139-174.ec2.internal\" DevicePath \"\"" Sep 10 13:44:21.704308 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:44:21.704251 2557 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d904068a-a5e6-402e-bc7a-9e048423f51a-workload-certs\") on node \"ip-10-0-139-174.ec2.internal\" DevicePath \"\"" Sep 10 13:44:21.704308 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:44:21.704264 2557 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d904068a-a5e6-402e-bc7a-9e048423f51a-credential-socket\") on node \"ip-10-0-139-174.ec2.internal\" DevicePath \"\"" Sep 10 13:44:21.705638 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:44:21.705616 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d904068a-a5e6-402e-bc7a-9e048423f51a-istio-token" (OuterVolumeSpecName: "istio-token") pod "d904068a-a5e6-402e-bc7a-9e048423f51a" (UID: "d904068a-a5e6-402e-bc7a-9e048423f51a"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 10 13:44:21.705704 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:44:21.705682 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/d904068a-a5e6-402e-bc7a-9e048423f51a-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "d904068a-a5e6-402e-bc7a-9e048423f51a" (UID: "d904068a-a5e6-402e-bc7a-9e048423f51a"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Sep 10 13:44:21.705910 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:44:21.705894 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d904068a-a5e6-402e-bc7a-9e048423f51a-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "d904068a-a5e6-402e-bc7a-9e048423f51a" (UID: "d904068a-a5e6-402e-bc7a-9e048423f51a"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 13:44:21.705910 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:44:21.705904 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d904068a-a5e6-402e-bc7a-9e048423f51a-kube-api-access-slsfj" (OuterVolumeSpecName: "kube-api-access-slsfj") pod "d904068a-a5e6-402e-bc7a-9e048423f51a" (UID: "d904068a-a5e6-402e-bc7a-9e048423f51a"). InnerVolumeSpecName "kube-api-access-slsfj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 10 13:44:21.804216 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:44:21.804187 2557 generic.go:356] "Generic (PLEG): container finished" podID="d904068a-a5e6-402e-bc7a-9e048423f51a" containerID="ac0bdcb66dc8bd3e36c25453383751ce99223f3b0876e537e4e66d2f0e2efbf0" exitCode=0 Sep 10 13:44:21.804320 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:44:21.804234 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-cleanup-37280b-openshift-default-769cf8bfd8-2xj6w" event={"ID":"d904068a-a5e6-402e-bc7a-9e048423f51a","Type":"ContainerDied","Data":"ac0bdcb66dc8bd3e36c25453383751ce99223f3b0876e537e4e66d2f0e2efbf0"} Sep 10 13:44:21.804320 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:44:21.804245 2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-cleanup-37280b-openshift-default-769cf8bfd8-2xj6w" Sep 10 13:44:21.804320 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:44:21.804287 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-cleanup-37280b-openshift-default-769cf8bfd8-2xj6w" event={"ID":"d904068a-a5e6-402e-bc7a-9e048423f51a","Type":"ContainerDied","Data":"06021632d4ab1ba25b11010960e32be9483e47b42babd1f7dba844641ae8b5ab"} Sep 10 13:44:21.804320 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:44:21.804308 2557 scope.go:117] "RemoveContainer" containerID="ac0bdcb66dc8bd3e36c25453383751ce99223f3b0876e537e4e66d2f0e2efbf0" Sep 10 13:44:21.804854 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:44:21.804826 2557 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-slsfj\" (UniqueName: \"kubernetes.io/projected/d904068a-a5e6-402e-bc7a-9e048423f51a-kube-api-access-slsfj\") on node \"ip-10-0-139-174.ec2.internal\" DevicePath \"\"" Sep 10 13:44:21.804854 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:44:21.804850 2557 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d904068a-a5e6-402e-bc7a-9e048423f51a-istio-podinfo\") on node \"ip-10-0-139-174.ec2.internal\" DevicePath \"\"" Sep 10 13:44:21.804999 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:44:21.804864 2557 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d904068a-a5e6-402e-bc7a-9e048423f51a-istio-token\") on node \"ip-10-0-139-174.ec2.internal\" DevicePath \"\"" Sep 10 13:44:21.804999 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:44:21.804876 2557 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d904068a-a5e6-402e-bc7a-9e048423f51a-istio-envoy\") on node \"ip-10-0-139-174.ec2.internal\" DevicePath \"\"" Sep 10 13:44:21.806260 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:44:21.806237 2557 generic.go:356] "Generic (PLEG): container finished" podID="78c88d9c-e196-4446-99eb-7d15f395077a" containerID="35fdb6de4b42903ec326366d0582c5931b3b3e55f37add1710a1984ff669d88b" exitCode=0 Sep 10 13:44:21.806341 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:44:21.806310 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-56b405-wm429" event={"ID":"78c88d9c-e196-4446-99eb-7d15f395077a","Type":"ContainerDied","Data":"35fdb6de4b42903ec326366d0582c5931b3b3e55f37add1710a1984ff669d88b"} Sep 10 13:44:21.806403 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:44:21.806349 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-56b405-wm429" event={"ID":"78c88d9c-e196-4446-99eb-7d15f395077a","Type":"ContainerStarted","Data":"828627d290e20cd6054eab58be606edb97688c31b9cd5d6ac1e52bdf2292cb76"} Sep 10 13:44:21.814526 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:44:21.814509 2557 scope.go:117] "RemoveContainer" containerID="9fd58b379d987eb2dd4b7a9a3329da675117f8fe4ef36b1bfd7528965f27726b" Sep 10 13:44:21.822836 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:44:21.822820 2557 scope.go:117] "RemoveContainer" containerID="ac0bdcb66dc8bd3e36c25453383751ce99223f3b0876e537e4e66d2f0e2efbf0" Sep 10 13:44:21.823085 ip-10-0-139-174 kubenswrapper[2557]: E0910 13:44:21.823068 2557 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ac0bdcb66dc8bd3e36c25453383751ce99223f3b0876e537e4e66d2f0e2efbf0\": container with ID starting with ac0bdcb66dc8bd3e36c25453383751ce99223f3b0876e537e4e66d2f0e2efbf0 not found: ID does not exist" containerID="ac0bdcb66dc8bd3e36c25453383751ce99223f3b0876e537e4e66d2f0e2efbf0" Sep 10 13:44:21.823125 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:44:21.823094 2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ac0bdcb66dc8bd3e36c25453383751ce99223f3b0876e537e4e66d2f0e2efbf0"} err="failed to get container status \"ac0bdcb66dc8bd3e36c25453383751ce99223f3b0876e537e4e66d2f0e2efbf0\": rpc error: code = NotFound desc = could not find container \"ac0bdcb66dc8bd3e36c25453383751ce99223f3b0876e537e4e66d2f0e2efbf0\": container with ID starting with ac0bdcb66dc8bd3e36c25453383751ce99223f3b0876e537e4e66d2f0e2efbf0 not found: ID does not exist" Sep 10 13:44:21.823125 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:44:21.823110 2557 scope.go:117] "RemoveContainer" containerID="9fd58b379d987eb2dd4b7a9a3329da675117f8fe4ef36b1bfd7528965f27726b" Sep 10 13:44:21.823348 ip-10-0-139-174 kubenswrapper[2557]: E0910 13:44:21.823335 2557 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9fd58b379d987eb2dd4b7a9a3329da675117f8fe4ef36b1bfd7528965f27726b\": container with ID starting with 9fd58b379d987eb2dd4b7a9a3329da675117f8fe4ef36b1bfd7528965f27726b not found: ID does not exist" containerID="9fd58b379d987eb2dd4b7a9a3329da675117f8fe4ef36b1bfd7528965f27726b" Sep 10 13:44:21.823388 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:44:21.823351 2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9fd58b379d987eb2dd4b7a9a3329da675117f8fe4ef36b1bfd7528965f27726b"} err="failed to get container status \"9fd58b379d987eb2dd4b7a9a3329da675117f8fe4ef36b1bfd7528965f27726b\": rpc error: code = NotFound desc = could not find container \"9fd58b379d987eb2dd4b7a9a3329da675117f8fe4ef36b1bfd7528965f27726b\": container with ID starting with 9fd58b379d987eb2dd4b7a9a3329da675117f8fe4ef36b1bfd7528965f27726b not found: ID does not exist" Sep 10 13:44:21.854077 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:44:21.854052 2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-cleanup-37280b-openshift-default-769cf8bfd8-2xj6w"] Sep 10 13:44:21.860804 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:44:21.860785 2557 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-cleanup-37280b-openshift-default-769cf8bfd8-2xj6w"] Sep 10 13:44:22.220667 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:44:22.220645 2557 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d904068a-a5e6-402e-bc7a-9e048423f51a" path="/var/lib/kubelet/pods/d904068a-a5e6-402e-bc7a-9e048423f51a/volumes" Sep 10 13:44:22.943162 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:44:22.943137 2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-56b405-wm429" Sep 10 13:44:23.114584 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:44:23.114510 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/78c88d9c-e196-4446-99eb-7d15f395077a-maas-api-service-ca\") pod \"78c88d9c-e196-4446-99eb-7d15f395077a\" (UID: \"78c88d9c-e196-4446-99eb-7d15f395077a\") " Sep 10 13:44:23.114827 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:44:23.114807 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/78c88d9c-e196-4446-99eb-7d15f395077a-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "78c88d9c-e196-4446-99eb-7d15f395077a" (UID: "78c88d9c-e196-4446-99eb-7d15f395077a"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 10 13:44:23.215873 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:44:23.215846 2557 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/78c88d9c-e196-4446-99eb-7d15f395077a-maas-api-service-ca\") on node \"ip-10-0-139-174.ec2.internal\" DevicePath \"\"" Sep 10 13:44:23.819531 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:44:23.819506 2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-56b405-wm429" Sep 10 13:44:23.819754 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:44:23.819548 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-56b405-wm429" event={"ID":"78c88d9c-e196-4446-99eb-7d15f395077a","Type":"ContainerDied","Data":"828627d290e20cd6054eab58be606edb97688c31b9cd5d6ac1e52bdf2292cb76"} Sep 10 13:44:23.819754 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:44:23.819595 2557 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="828627d290e20cd6054eab58be606edb97688c31b9cd5d6ac1e52bdf2292cb76" Sep 10 13:44:36.358268 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:44:36.358233 2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-56b405-wm429"] Sep 10 13:44:36.363473 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:44:36.363450 2557 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-56b405-wm429"] Sep 10 13:44:38.219937 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:44:38.219903 2557 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="78c88d9c-e196-4446-99eb-7d15f395077a" path="/var/lib/kubelet/pods/78c88d9c-e196-4446-99eb-7d15f395077a/volumes" Sep 10 13:45:00.151875 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:45:00.151836 2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29817465-jkfgf"] Sep 10 13:45:00.152475 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:45:00.152415 2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d904068a-a5e6-402e-bc7a-9e048423f51a" containerName="istio-proxy" Sep 10 13:45:00.152475 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:45:00.152451 2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="d904068a-a5e6-402e-bc7a-9e048423f51a" containerName="istio-proxy" Sep 10 13:45:00.152619 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:45:00.152477 2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="78c88d9c-e196-4446-99eb-7d15f395077a" containerName="revoke-keys" Sep 10 13:45:00.152619 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:45:00.152487 2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="78c88d9c-e196-4446-99eb-7d15f395077a" containerName="revoke-keys" Sep 10 13:45:00.152619 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:45:00.152603 2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="78c88d9c-e196-4446-99eb-7d15f395077a" containerName="revoke-keys" Sep 10 13:45:00.152619 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:45:00.152618 2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="d904068a-a5e6-402e-bc7a-9e048423f51a" containerName="istio-proxy" Sep 10 13:45:00.152835 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:45:00.152626 2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="d904068a-a5e6-402e-bc7a-9e048423f51a" containerName="istio-proxy" Sep 10 13:45:00.158113 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:45:00.158091 2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-08b85c-29817465-rdfpw"] Sep 10 13:45:00.158256 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:45:00.158202 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29817465-jkfgf" Sep 10 13:45:00.158560 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:45:00.158538 2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d904068a-a5e6-402e-bc7a-9e048423f51a" containerName="istio-proxy" Sep 10 13:45:00.158560 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:45:00.158557 2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="d904068a-a5e6-402e-bc7a-9e048423f51a" containerName="istio-proxy" Sep 10 13:45:00.160563 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:45:00.160541 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-wtblv\"" Sep 10 13:45:00.161608 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:45:00.161593 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-08b85c-29817465-rdfpw" Sep 10 13:45:00.167480 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:45:00.167039 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29817465-jkfgf"] Sep 10 13:45:00.170854 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:45:00.170830 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-08b85c-29817465-rdfpw"] Sep 10 13:45:00.170968 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:45:00.170912 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29817465-jkfgf" Sep 10 13:45:00.180575 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:45:00.180554 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-08b85c-29817465-rdfpw" Sep 10 13:45:00.457564 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:45:00.457541 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29817465-jkfgf"] Sep 10 13:45:00.459005 ip-10-0-139-174 kubenswrapper[2557]: W0910 13:45:00.458976 2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod070db958_b00e_4b95_a0f5_751dbedf8833.slice/crio-557dd864a03138d7f718a5512ade74925cc7b6f6ecc94f37bc6fafd469dbc21e WatchSource:0}: Error finding container 557dd864a03138d7f718a5512ade74925cc7b6f6ecc94f37bc6fafd469dbc21e: Status 404 returned error can't find the container with id 557dd864a03138d7f718a5512ade74925cc7b6f6ecc94f37bc6fafd469dbc21e Sep 10 13:45:00.465335 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:45:00.465309 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-08b85c-29817465-rdfpw"] Sep 10 13:45:00.466813 ip-10-0-139-174 kubenswrapper[2557]: W0910 13:45:00.466786 2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd086ea6d_e88b_4a26_ba59_a71b96e99dcd.slice/crio-f868ef397c91c481fab45c316b187a9dffcb85adddfd2d69e331c0d1aaa135be WatchSource:0}: Error finding container f868ef397c91c481fab45c316b187a9dffcb85adddfd2d69e331c0d1aaa135be: Status 404 returned error can't find the container with id f868ef397c91c481fab45c316b187a9dffcb85adddfd2d69e331c0d1aaa135be Sep 10 13:45:01.040660 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:45:01.040569 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29817465-jkfgf" event={"ID":"070db958-b00e-4b95-a0f5-751dbedf8833","Type":"ContainerStarted","Data":"ec257aafc0a190767789f2f7dfb8b0d0250254e2335894d28bf0928bce69d229"} Sep 10 13:45:01.040660 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:45:01.040633 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29817465-jkfgf" event={"ID":"070db958-b00e-4b95-a0f5-751dbedf8833","Type":"ContainerStarted","Data":"557dd864a03138d7f718a5512ade74925cc7b6f6ecc94f37bc6fafd469dbc21e"} Sep 10 13:45:01.042439 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:45:01.042401 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-08b85c-29817465-rdfpw" event={"ID":"d086ea6d-e88b-4a26-ba59-a71b96e99dcd","Type":"ContainerStarted","Data":"ee4d9c10ad26efaff207cd37a4072f0baeb28f5ae539406966adce309de14ce7"} Sep 10 13:45:01.042535 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:45:01.042452 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-08b85c-29817465-rdfpw" event={"ID":"d086ea6d-e88b-4a26-ba59-a71b96e99dcd","Type":"ContainerStarted","Data":"f868ef397c91c481fab45c316b187a9dffcb85adddfd2d69e331c0d1aaa135be"} Sep 10 13:45:01.057350 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:45:01.057307 2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29817465-jkfgf" podStartSLOduration=1.057293148 podStartE2EDuration="1.057293148s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 13:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 13:45:01.055449776 +0000 UTC m=+1661.362444190" watchObservedRunningTime="2026-09-10 13:45:01.057293148 +0000 UTC m=+1661.364287560" Sep 10 13:45:01.075064 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:45:01.075024 2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-08b85c-29817465-rdfpw" podStartSLOduration=1.075010946 podStartE2EDuration="1.075010946s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 13:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 13:45:01.073282234 +0000 UTC m=+1661.380276647" watchObservedRunningTime="2026-09-10 13:45:01.075010946 +0000 UTC m=+1661.382005358" Sep 10 13:45:02.049320 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:45:02.049283 2557 generic.go:356] "Generic (PLEG): container finished" podID="d086ea6d-e88b-4a26-ba59-a71b96e99dcd" containerID="ee4d9c10ad26efaff207cd37a4072f0baeb28f5ae539406966adce309de14ce7" exitCode=0 Sep 10 13:45:02.049743 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:45:02.049376 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-08b85c-29817465-rdfpw" event={"ID":"d086ea6d-e88b-4a26-ba59-a71b96e99dcd","Type":"ContainerDied","Data":"ee4d9c10ad26efaff207cd37a4072f0baeb28f5ae539406966adce309de14ce7"} Sep 10 13:45:02.050972 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:45:02.050952 2557 generic.go:356] "Generic (PLEG): container finished" podID="070db958-b00e-4b95-a0f5-751dbedf8833" containerID="ec257aafc0a190767789f2f7dfb8b0d0250254e2335894d28bf0928bce69d229" exitCode=0 Sep 10 13:45:02.051090 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:45:02.051015 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29817465-jkfgf" event={"ID":"070db958-b00e-4b95-a0f5-751dbedf8833","Type":"ContainerDied","Data":"ec257aafc0a190767789f2f7dfb8b0d0250254e2335894d28bf0928bce69d229"} Sep 10 13:45:03.208759 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:45:03.208740 2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-08b85c-29817465-rdfpw" Sep 10 13:45:03.211457 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:45:03.211437 2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29817465-jkfgf" Sep 10 13:45:04.064353 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:45:04.064321 2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-08b85c-29817465-rdfpw" Sep 10 13:45:04.064539 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:45:04.064352 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-08b85c-29817465-rdfpw" event={"ID":"d086ea6d-e88b-4a26-ba59-a71b96e99dcd","Type":"ContainerDied","Data":"f868ef397c91c481fab45c316b187a9dffcb85adddfd2d69e331c0d1aaa135be"} Sep 10 13:45:04.064539 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:45:04.064397 2557 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f868ef397c91c481fab45c316b187a9dffcb85adddfd2d69e331c0d1aaa135be" Sep 10 13:45:04.065817 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:45:04.065796 2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29817465-jkfgf" Sep 10 13:45:04.065935 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:45:04.065831 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29817465-jkfgf" event={"ID":"070db958-b00e-4b95-a0f5-751dbedf8833","Type":"ContainerDied","Data":"557dd864a03138d7f718a5512ade74925cc7b6f6ecc94f37bc6fafd469dbc21e"} Sep 10 13:45:04.065935 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:45:04.065865 2557 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="557dd864a03138d7f718a5512ade74925cc7b6f6ecc94f37bc6fafd469dbc21e" Sep 10 13:45:26.816599 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:45:26.816567 2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-08b85c-tbnjz"] Sep 10 13:45:26.816960 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:45:26.816950 2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d086ea6d-e88b-4a26-ba59-a71b96e99dcd" containerName="cleanup" Sep 10 13:45:26.816999 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:45:26.816962 2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="d086ea6d-e88b-4a26-ba59-a71b96e99dcd" containerName="cleanup" Sep 10 13:45:26.816999 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:45:26.816978 2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="070db958-b00e-4b95-a0f5-751dbedf8833" containerName="cleanup" Sep 10 13:45:26.816999 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:45:26.816983 2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="070db958-b00e-4b95-a0f5-751dbedf8833" containerName="cleanup" Sep 10 13:45:26.817093 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:45:26.817052 2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="d086ea6d-e88b-4a26-ba59-a71b96e99dcd" containerName="cleanup" Sep 10 13:45:26.817093 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:45:26.817060 2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="070db958-b00e-4b95-a0f5-751dbedf8833" containerName="cleanup" Sep 10 13:45:26.821225 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:45:26.821205 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-08b85c-tbnjz" Sep 10 13:45:26.825542 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:45:26.825525 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-wtblv\"" Sep 10 13:45:26.826193 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:45:26.826163 2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Sep 10 13:45:26.839354 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:45:26.839330 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-08b85c-tbnjz"] Sep 10 13:45:26.951430 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:45:26.951399 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2b934432-90fb-4c30-918e-157a9e927d2c-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-08b85c-tbnjz\" (UID: \"2b934432-90fb-4c30-918e-157a9e927d2c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-08b85c-tbnjz" Sep 10 13:45:27.052782 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:45:27.052734 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2b934432-90fb-4c30-918e-157a9e927d2c-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-08b85c-tbnjz\" (UID: \"2b934432-90fb-4c30-918e-157a9e927d2c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-08b85c-tbnjz" Sep 10 13:45:27.053686 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:45:27.053647 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2b934432-90fb-4c30-918e-157a9e927d2c-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-08b85c-tbnjz\" (UID: \"2b934432-90fb-4c30-918e-157a9e927d2c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-08b85c-tbnjz" Sep 10 13:45:27.131468 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:45:27.131365 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-08b85c-tbnjz" Sep 10 13:45:27.416392 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:45:27.416366 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-08b85c-tbnjz"] Sep 10 13:45:27.418571 ip-10-0-139-174 kubenswrapper[2557]: W0910 13:45:27.418541 2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2b934432_90fb_4c30_918e_157a9e927d2c.slice/crio-dc767d2b426ded5705ee9115ffc0c42636bcba1c1837ee213a898289717d2b9a WatchSource:0}: Error finding container dc767d2b426ded5705ee9115ffc0c42636bcba1c1837ee213a898289717d2b9a: Status 404 returned error can't find the container with id dc767d2b426ded5705ee9115ffc0c42636bcba1c1837ee213a898289717d2b9a Sep 10 13:45:28.208166 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:45:28.208137 2557 generic.go:356] "Generic (PLEG): container finished" podID="2b934432-90fb-4c30-918e-157a9e927d2c" containerID="bfa12f90b9a3d411e24ebfa22eccbc75d3e14b860dbd0a1327dbf7c8a72b9668" exitCode=0 Sep 10 13:45:28.208537 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:45:28.208231 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-08b85c-tbnjz" event={"ID":"2b934432-90fb-4c30-918e-157a9e927d2c","Type":"ContainerDied","Data":"bfa12f90b9a3d411e24ebfa22eccbc75d3e14b860dbd0a1327dbf7c8a72b9668"} Sep 10 13:45:28.208537 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:45:28.208289 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-08b85c-tbnjz" event={"ID":"2b934432-90fb-4c30-918e-157a9e927d2c","Type":"ContainerStarted","Data":"dc767d2b426ded5705ee9115ffc0c42636bcba1c1837ee213a898289717d2b9a"} Sep 10 13:45:29.340958 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:45:29.340928 2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-08b85c-tbnjz" Sep 10 13:45:29.474090 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:45:29.474004 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2b934432-90fb-4c30-918e-157a9e927d2c-maas-api-service-ca\") pod \"2b934432-90fb-4c30-918e-157a9e927d2c\" (UID: \"2b934432-90fb-4c30-918e-157a9e927d2c\") " Sep 10 13:45:29.474312 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:45:29.474292 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2b934432-90fb-4c30-918e-157a9e927d2c-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "2b934432-90fb-4c30-918e-157a9e927d2c" (UID: "2b934432-90fb-4c30-918e-157a9e927d2c"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 10 13:45:29.574701 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:45:29.574675 2557 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2b934432-90fb-4c30-918e-157a9e927d2c-maas-api-service-ca\") on node \"ip-10-0-139-174.ec2.internal\" DevicePath \"\"" Sep 10 13:45:30.222165 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:45:30.222140 2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-08b85c-tbnjz" Sep 10 13:45:30.222350 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:45:30.222181 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-08b85c-tbnjz" event={"ID":"2b934432-90fb-4c30-918e-157a9e927d2c","Type":"ContainerDied","Data":"dc767d2b426ded5705ee9115ffc0c42636bcba1c1837ee213a898289717d2b9a"} Sep 10 13:45:30.222350 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:45:30.222231 2557 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="dc767d2b426ded5705ee9115ffc0c42636bcba1c1837ee213a898289717d2b9a" Sep 10 13:45:37.510811 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:45:37.510772 2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-08b85c-29817465-rdfpw"] Sep 10 13:45:37.513306 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:45:37.513277 2557 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-08b85c-29817465-rdfpw"] Sep 10 13:45:37.542164 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:45:37.542137 2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-08b85c-5cc85974f5-ncbzk"] Sep 10 13:45:37.542645 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:45:37.542612 2557 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-b-08b85c-5cc85974f5-ncbzk" podUID="f676c0de-5f1d-4ebc-aa86-6de59306b195" containerName="payload-processing" containerID="cri-o://0273e594d5ee7889342dbc48d39ddc4497b5296e182133538f490fdef44b3dbf" gracePeriod=30 Sep 10 13:45:37.756056 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:45:37.756028 2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-08b85c-tbnjz"] Sep 10 13:45:37.758449 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:45:37.758409 2557 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-08b85c-tbnjz"] Sep 10 13:45:37.793194 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:45:37.793169 2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-08b85c-5cc85974f5-ncbzk" Sep 10 13:45:37.948547 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:45:37.948518 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-tqwvj\" (UniqueName: \"kubernetes.io/projected/f676c0de-5f1d-4ebc-aa86-6de59306b195-kube-api-access-tqwvj\") pod \"f676c0de-5f1d-4ebc-aa86-6de59306b195\" (UID: \"f676c0de-5f1d-4ebc-aa86-6de59306b195\") " Sep 10 13:45:37.948740 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:45:37.948616 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f676c0de-5f1d-4ebc-aa86-6de59306b195-plugins-config-volume\") pod \"f676c0de-5f1d-4ebc-aa86-6de59306b195\" (UID: \"f676c0de-5f1d-4ebc-aa86-6de59306b195\") " Sep 10 13:45:37.948964 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:45:37.948939 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f676c0de-5f1d-4ebc-aa86-6de59306b195-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "f676c0de-5f1d-4ebc-aa86-6de59306b195" (UID: "f676c0de-5f1d-4ebc-aa86-6de59306b195"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 10 13:45:37.950781 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:45:37.950753 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f676c0de-5f1d-4ebc-aa86-6de59306b195-kube-api-access-tqwvj" (OuterVolumeSpecName: "kube-api-access-tqwvj") pod "f676c0de-5f1d-4ebc-aa86-6de59306b195" (UID: "f676c0de-5f1d-4ebc-aa86-6de59306b195"). InnerVolumeSpecName "kube-api-access-tqwvj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 10 13:45:38.050244 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:45:38.050173 2557 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tqwvj\" (UniqueName: \"kubernetes.io/projected/f676c0de-5f1d-4ebc-aa86-6de59306b195-kube-api-access-tqwvj\") on node \"ip-10-0-139-174.ec2.internal\" DevicePath \"\"" Sep 10 13:45:38.050244 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:45:38.050201 2557 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f676c0de-5f1d-4ebc-aa86-6de59306b195-plugins-config-volume\") on node \"ip-10-0-139-174.ec2.internal\" DevicePath \"\"" Sep 10 13:45:38.221736 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:45:38.221706 2557 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2b934432-90fb-4c30-918e-157a9e927d2c" path="/var/lib/kubelet/pods/2b934432-90fb-4c30-918e-157a9e927d2c/volumes" Sep 10 13:45:38.222056 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:45:38.222040 2557 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d086ea6d-e88b-4a26-ba59-a71b96e99dcd" path="/var/lib/kubelet/pods/d086ea6d-e88b-4a26-ba59-a71b96e99dcd/volumes" Sep 10 13:45:38.271268 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:45:38.271236 2557 generic.go:356] "Generic (PLEG): container finished" podID="f676c0de-5f1d-4ebc-aa86-6de59306b195" containerID="0273e594d5ee7889342dbc48d39ddc4497b5296e182133538f490fdef44b3dbf" exitCode=0 Sep 10 13:45:38.271450 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:45:38.271282 2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-08b85c-5cc85974f5-ncbzk" Sep 10 13:45:38.271450 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:45:38.271280 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-08b85c-5cc85974f5-ncbzk" event={"ID":"f676c0de-5f1d-4ebc-aa86-6de59306b195","Type":"ContainerDied","Data":"0273e594d5ee7889342dbc48d39ddc4497b5296e182133538f490fdef44b3dbf"} Sep 10 13:45:38.271450 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:45:38.271345 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-08b85c-5cc85974f5-ncbzk" event={"ID":"f676c0de-5f1d-4ebc-aa86-6de59306b195","Type":"ContainerDied","Data":"a0afbc0371f31eca5b506acfa5036ae4b5679d633f86b188d11ea6872f32e85d"} Sep 10 13:45:38.271450 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:45:38.271379 2557 scope.go:117] "RemoveContainer" containerID="0273e594d5ee7889342dbc48d39ddc4497b5296e182133538f490fdef44b3dbf" Sep 10 13:45:38.280366 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:45:38.280351 2557 scope.go:117] "RemoveContainer" containerID="0273e594d5ee7889342dbc48d39ddc4497b5296e182133538f490fdef44b3dbf" Sep 10 13:45:38.280666 ip-10-0-139-174 kubenswrapper[2557]: E0910 13:45:38.280646 2557 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0273e594d5ee7889342dbc48d39ddc4497b5296e182133538f490fdef44b3dbf\": container with ID starting with 0273e594d5ee7889342dbc48d39ddc4497b5296e182133538f490fdef44b3dbf not found: ID does not exist" containerID="0273e594d5ee7889342dbc48d39ddc4497b5296e182133538f490fdef44b3dbf" Sep 10 13:45:38.280724 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:45:38.280675 2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0273e594d5ee7889342dbc48d39ddc4497b5296e182133538f490fdef44b3dbf"} err="failed to get container status \"0273e594d5ee7889342dbc48d39ddc4497b5296e182133538f490fdef44b3dbf\": rpc error: code = NotFound desc = could not find container \"0273e594d5ee7889342dbc48d39ddc4497b5296e182133538f490fdef44b3dbf\": container with ID starting with 0273e594d5ee7889342dbc48d39ddc4497b5296e182133538f490fdef44b3dbf not found: ID does not exist" Sep 10 13:45:38.306119 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:45:38.306066 2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-08b85c-5cc85974f5-ncbzk"] Sep 10 13:45:38.319899 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:45:38.319870 2557 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-08b85c-5cc85974f5-ncbzk"] Sep 10 13:45:40.221225 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:45:40.221194 2557 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f676c0de-5f1d-4ebc-aa86-6de59306b195" path="/var/lib/kubelet/pods/f676c0de-5f1d-4ebc-aa86-6de59306b195/volumes" Sep 10 13:45:45.524252 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:45:45.524171 2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-7657ef-hqsz7"] Sep 10 13:45:45.524628 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:45:45.524565 2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2b934432-90fb-4c30-918e-157a9e927d2c" containerName="revoke-keys" Sep 10 13:45:45.524628 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:45:45.524577 2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="2b934432-90fb-4c30-918e-157a9e927d2c" containerName="revoke-keys" Sep 10 13:45:45.524628 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:45:45.524595 2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f676c0de-5f1d-4ebc-aa86-6de59306b195" containerName="payload-processing" Sep 10 13:45:45.524628 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:45:45.524601 2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="f676c0de-5f1d-4ebc-aa86-6de59306b195" containerName="payload-processing" Sep 10 13:45:45.524768 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:45:45.524658 2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="f676c0de-5f1d-4ebc-aa86-6de59306b195" containerName="payload-processing" Sep 10 13:45:45.524768 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:45:45.524668 2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="2b934432-90fb-4c30-918e-157a9e927d2c" containerName="revoke-keys" Sep 10 13:45:45.529063 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:45:45.529045 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-7657ef-hqsz7" Sep 10 13:45:45.531403 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:45:45.531383 2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Sep 10 13:45:45.534308 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:45:45.534147 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-wtblv\"" Sep 10 13:45:45.537572 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:45:45.537545 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-7657ef-hqsz7"] Sep 10 13:45:45.606261 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:45:45.606229 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/94de69cd-36ae-4d1e-86c7-2c3b09e8f3f2-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w3-7657ef-hqsz7\" (UID: \"94de69cd-36ae-4d1e-86c7-2c3b09e8f3f2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-7657ef-hqsz7" Sep 10 13:45:45.707541 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:45:45.707509 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/94de69cd-36ae-4d1e-86c7-2c3b09e8f3f2-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w3-7657ef-hqsz7\" (UID: \"94de69cd-36ae-4d1e-86c7-2c3b09e8f3f2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-7657ef-hqsz7" Sep 10 13:45:45.708140 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:45:45.708121 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/94de69cd-36ae-4d1e-86c7-2c3b09e8f3f2-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w3-7657ef-hqsz7\" (UID: \"94de69cd-36ae-4d1e-86c7-2c3b09e8f3f2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-7657ef-hqsz7" Sep 10 13:45:45.841614 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:45:45.841540 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-7657ef-hqsz7" Sep 10 13:45:46.124343 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:45:46.124307 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-7657ef-hqsz7"] Sep 10 13:45:46.125230 ip-10-0-139-174 kubenswrapper[2557]: W0910 13:45:46.125202 2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod94de69cd_36ae_4d1e_86c7_2c3b09e8f3f2.slice/crio-d66a4d93e4fbf9803f0669ed4c0c7ed71229af58b1f9a5883b33ac8664a60420 WatchSource:0}: Error finding container d66a4d93e4fbf9803f0669ed4c0c7ed71229af58b1f9a5883b33ac8664a60420: Status 404 returned error can't find the container with id d66a4d93e4fbf9803f0669ed4c0c7ed71229af58b1f9a5883b33ac8664a60420 Sep 10 13:45:46.325086 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:45:46.325053 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-7657ef-hqsz7" event={"ID":"94de69cd-36ae-4d1e-86c7-2c3b09e8f3f2","Type":"ContainerStarted","Data":"d66a4d93e4fbf9803f0669ed4c0c7ed71229af58b1f9a5883b33ac8664a60420"} Sep 10 13:45:47.333071 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:45:47.333040 2557 generic.go:356] "Generic (PLEG): container finished" podID="94de69cd-36ae-4d1e-86c7-2c3b09e8f3f2" containerID="03c9b9dc0293970d73551f6477668cb378a778d21c0f552fb48868c339c5e569" exitCode=0 Sep 10 13:45:47.333466 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:45:47.333096 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-7657ef-hqsz7" event={"ID":"94de69cd-36ae-4d1e-86c7-2c3b09e8f3f2","Type":"ContainerDied","Data":"03c9b9dc0293970d73551f6477668cb378a778d21c0f552fb48868c339c5e569"} Sep 10 13:45:48.465776 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:45:48.465750 2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-7657ef-hqsz7" Sep 10 13:45:48.534875 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:45:48.534848 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/94de69cd-36ae-4d1e-86c7-2c3b09e8f3f2-maas-api-service-ca\") pod \"94de69cd-36ae-4d1e-86c7-2c3b09e8f3f2\" (UID: \"94de69cd-36ae-4d1e-86c7-2c3b09e8f3f2\") " Sep 10 13:45:48.535226 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:45:48.535205 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/94de69cd-36ae-4d1e-86c7-2c3b09e8f3f2-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "94de69cd-36ae-4d1e-86c7-2c3b09e8f3f2" (UID: "94de69cd-36ae-4d1e-86c7-2c3b09e8f3f2"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 10 13:45:48.636627 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:45:48.636545 2557 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/94de69cd-36ae-4d1e-86c7-2c3b09e8f3f2-maas-api-service-ca\") on node \"ip-10-0-139-174.ec2.internal\" DevicePath \"\"" Sep 10 13:45:49.346055 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:45:49.346026 2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-7657ef-hqsz7" Sep 10 13:45:49.346234 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:45:49.346060 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-7657ef-hqsz7" event={"ID":"94de69cd-36ae-4d1e-86c7-2c3b09e8f3f2","Type":"ContainerDied","Data":"d66a4d93e4fbf9803f0669ed4c0c7ed71229af58b1f9a5883b33ac8664a60420"} Sep 10 13:45:49.346234 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:45:49.346101 2557 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d66a4d93e4fbf9803f0669ed4c0c7ed71229af58b1f9a5883b33ac8664a60420" Sep 10 13:45:57.033432 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:45:57.033381 2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-7657ef-hqsz7"] Sep 10 13:45:57.042316 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:45:57.042294 2557 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-7657ef-hqsz7"] Sep 10 13:45:58.219638 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:45:58.219605 2557 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="94de69cd-36ae-4d1e-86c7-2c3b09e8f3f2" path="/var/lib/kubelet/pods/94de69cd-36ae-4d1e-86c7-2c3b09e8f3f2/volumes" Sep 10 13:46:05.636266 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:46:05.636230 2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-7657ef-openshift-default-85d55f76f6-gw88p"] Sep 10 13:46:05.636728 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:46:05.636478 2557 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w3-7657ef-openshift-default-85d55f76f6-gw88p" podUID="979718af-bb04-448e-8c47-4c82a9adce63" containerName="istio-proxy" containerID="cri-o://ad663552ea2d18c5a57c99e5eabe50898a9deceaa2491c336b626268f58d9acb" gracePeriod=30 Sep 10 13:46:10.881928 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:46:10.881904 2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w3-7657ef-openshift-default-85d55f76f6-gw88p" Sep 10 13:46:11.030042 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:46:11.030012 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/979718af-bb04-448e-8c47-4c82a9adce63-credential-socket\") pod \"979718af-bb04-448e-8c47-4c82a9adce63\" (UID: \"979718af-bb04-448e-8c47-4c82a9adce63\") " Sep 10 13:46:11.030190 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:46:11.030048 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/979718af-bb04-448e-8c47-4c82a9adce63-workload-certs\") pod \"979718af-bb04-448e-8c47-4c82a9adce63\" (UID: \"979718af-bb04-448e-8c47-4c82a9adce63\") " Sep 10 13:46:11.030190 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:46:11.030075 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/979718af-bb04-448e-8c47-4c82a9adce63-istio-data\") pod \"979718af-bb04-448e-8c47-4c82a9adce63\" (UID: \"979718af-bb04-448e-8c47-4c82a9adce63\") " Sep 10 13:46:11.030190 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:46:11.030098 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6k2jp\" (UniqueName: \"kubernetes.io/projected/979718af-bb04-448e-8c47-4c82a9adce63-kube-api-access-6k2jp\") pod \"979718af-bb04-448e-8c47-4c82a9adce63\" (UID: \"979718af-bb04-448e-8c47-4c82a9adce63\") " Sep 10 13:46:11.030190 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:46:11.030138 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/979718af-bb04-448e-8c47-4c82a9adce63-istio-podinfo\") pod \"979718af-bb04-448e-8c47-4c82a9adce63\" (UID: \"979718af-bb04-448e-8c47-4c82a9adce63\") " Sep 10 13:46:11.030190 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:46:11.030169 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/979718af-bb04-448e-8c47-4c82a9adce63-istio-token\") pod \"979718af-bb04-448e-8c47-4c82a9adce63\" (UID: \"979718af-bb04-448e-8c47-4c82a9adce63\") " Sep 10 13:46:11.030449 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:46:11.030219 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/979718af-bb04-448e-8c47-4c82a9adce63-istio-envoy\") pod \"979718af-bb04-448e-8c47-4c82a9adce63\" (UID: \"979718af-bb04-448e-8c47-4c82a9adce63\") " Sep 10 13:46:11.030449 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:46:11.030244 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/979718af-bb04-448e-8c47-4c82a9adce63-workload-socket\") pod \"979718af-bb04-448e-8c47-4c82a9adce63\" (UID: \"979718af-bb04-448e-8c47-4c82a9adce63\") " Sep 10 13:46:11.030449 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:46:11.030306 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/979718af-bb04-448e-8c47-4c82a9adce63-istiod-ca-cert\") pod \"979718af-bb04-448e-8c47-4c82a9adce63\" (UID: \"979718af-bb04-448e-8c47-4c82a9adce63\") " Sep 10 13:46:11.030449 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:46:11.030318 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/979718af-bb04-448e-8c47-4c82a9adce63-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "979718af-bb04-448e-8c47-4c82a9adce63" (UID: "979718af-bb04-448e-8c47-4c82a9adce63"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 13:46:11.030449 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:46:11.030333 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/979718af-bb04-448e-8c47-4c82a9adce63-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "979718af-bb04-448e-8c47-4c82a9adce63" (UID: "979718af-bb04-448e-8c47-4c82a9adce63"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 13:46:11.030449 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:46:11.030381 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/979718af-bb04-448e-8c47-4c82a9adce63-istio-data" (OuterVolumeSpecName: "istio-data") pod "979718af-bb04-448e-8c47-4c82a9adce63" (UID: "979718af-bb04-448e-8c47-4c82a9adce63"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 13:46:11.030724 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:46:11.030587 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/979718af-bb04-448e-8c47-4c82a9adce63-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "979718af-bb04-448e-8c47-4c82a9adce63" (UID: "979718af-bb04-448e-8c47-4c82a9adce63"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 13:46:11.030724 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:46:11.030714 2557 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/979718af-bb04-448e-8c47-4c82a9adce63-credential-socket\") on node \"ip-10-0-139-174.ec2.internal\" DevicePath \"\"" Sep 10 13:46:11.030831 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:46:11.030727 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/979718af-bb04-448e-8c47-4c82a9adce63-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "979718af-bb04-448e-8c47-4c82a9adce63" (UID: "979718af-bb04-448e-8c47-4c82a9adce63"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 10 13:46:11.030831 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:46:11.030734 2557 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/979718af-bb04-448e-8c47-4c82a9adce63-workload-certs\") on node \"ip-10-0-139-174.ec2.internal\" DevicePath \"\"" Sep 10 13:46:11.030831 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:46:11.030748 2557 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/979718af-bb04-448e-8c47-4c82a9adce63-istio-data\") on node \"ip-10-0-139-174.ec2.internal\" DevicePath \"\"" Sep 10 13:46:11.030831 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:46:11.030758 2557 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/979718af-bb04-448e-8c47-4c82a9adce63-workload-socket\") on node \"ip-10-0-139-174.ec2.internal\" DevicePath \"\"" Sep 10 13:46:11.032542 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:46:11.032514 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/979718af-bb04-448e-8c47-4c82a9adce63-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "979718af-bb04-448e-8c47-4c82a9adce63" (UID: "979718af-bb04-448e-8c47-4c82a9adce63"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 13:46:11.032542 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:46:11.032520 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/979718af-bb04-448e-8c47-4c82a9adce63-istio-token" (OuterVolumeSpecName: "istio-token") pod "979718af-bb04-448e-8c47-4c82a9adce63" (UID: "979718af-bb04-448e-8c47-4c82a9adce63"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 10 13:46:11.032876 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:46:11.032856 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/979718af-bb04-448e-8c47-4c82a9adce63-kube-api-access-6k2jp" (OuterVolumeSpecName: "kube-api-access-6k2jp") pod "979718af-bb04-448e-8c47-4c82a9adce63" (UID: "979718af-bb04-448e-8c47-4c82a9adce63"). InnerVolumeSpecName "kube-api-access-6k2jp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 10 13:46:11.032922 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:46:11.032867 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/979718af-bb04-448e-8c47-4c82a9adce63-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "979718af-bb04-448e-8c47-4c82a9adce63" (UID: "979718af-bb04-448e-8c47-4c82a9adce63"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Sep 10 13:46:11.131690 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:46:11.131664 2557 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6k2jp\" (UniqueName: \"kubernetes.io/projected/979718af-bb04-448e-8c47-4c82a9adce63-kube-api-access-6k2jp\") on node \"ip-10-0-139-174.ec2.internal\" DevicePath \"\"" Sep 10 13:46:11.131690 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:46:11.131684 2557 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/979718af-bb04-448e-8c47-4c82a9adce63-istio-podinfo\") on node \"ip-10-0-139-174.ec2.internal\" DevicePath \"\"" Sep 10 13:46:11.131690 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:46:11.131692 2557 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/979718af-bb04-448e-8c47-4c82a9adce63-istio-token\") on node \"ip-10-0-139-174.ec2.internal\" DevicePath \"\"" Sep 10 13:46:11.131879 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:46:11.131699 2557 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/979718af-bb04-448e-8c47-4c82a9adce63-istio-envoy\") on node \"ip-10-0-139-174.ec2.internal\" DevicePath \"\"" Sep 10 13:46:11.131879 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:46:11.131715 2557 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/979718af-bb04-448e-8c47-4c82a9adce63-istiod-ca-cert\") on node \"ip-10-0-139-174.ec2.internal\" DevicePath \"\"" Sep 10 13:46:11.477564 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:46:11.477531 2557 generic.go:356] "Generic (PLEG): container finished" podID="979718af-bb04-448e-8c47-4c82a9adce63" containerID="ad663552ea2d18c5a57c99e5eabe50898a9deceaa2491c336b626268f58d9acb" exitCode=0 Sep 10 13:46:11.477729 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:46:11.477577 2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w3-7657ef-openshift-default-85d55f76f6-gw88p" Sep 10 13:46:11.477729 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:46:11.477622 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-7657ef-openshift-default-85d55f76f6-gw88p" event={"ID":"979718af-bb04-448e-8c47-4c82a9adce63","Type":"ContainerDied","Data":"ad663552ea2d18c5a57c99e5eabe50898a9deceaa2491c336b626268f58d9acb"} Sep 10 13:46:11.477729 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:46:11.477678 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-7657ef-openshift-default-85d55f76f6-gw88p" event={"ID":"979718af-bb04-448e-8c47-4c82a9adce63","Type":"ContainerDied","Data":"34ca8ce9cde3717101e20496d77e051311fa6e34f2de5651fc38815ac6c8a63a"} Sep 10 13:46:11.477729 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:46:11.477705 2557 scope.go:117] "RemoveContainer" containerID="ad663552ea2d18c5a57c99e5eabe50898a9deceaa2491c336b626268f58d9acb" Sep 10 13:46:11.487359 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:46:11.487338 2557 scope.go:117] "RemoveContainer" containerID="c9bc7d0ceed89ce9f9cc0ad949f01ea822c607e00482217d447fb53735bfdb0e" Sep 10 13:46:11.495794 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:46:11.495775 2557 scope.go:117] "RemoveContainer" containerID="ad663552ea2d18c5a57c99e5eabe50898a9deceaa2491c336b626268f58d9acb" Sep 10 13:46:11.496041 ip-10-0-139-174 kubenswrapper[2557]: E0910 13:46:11.496025 2557 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ad663552ea2d18c5a57c99e5eabe50898a9deceaa2491c336b626268f58d9acb\": container with ID starting with ad663552ea2d18c5a57c99e5eabe50898a9deceaa2491c336b626268f58d9acb not found: ID does not exist" containerID="ad663552ea2d18c5a57c99e5eabe50898a9deceaa2491c336b626268f58d9acb" Sep 10 13:46:11.496095 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:46:11.496048 2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ad663552ea2d18c5a57c99e5eabe50898a9deceaa2491c336b626268f58d9acb"} err="failed to get container status \"ad663552ea2d18c5a57c99e5eabe50898a9deceaa2491c336b626268f58d9acb\": rpc error: code = NotFound desc = could not find container \"ad663552ea2d18c5a57c99e5eabe50898a9deceaa2491c336b626268f58d9acb\": container with ID starting with ad663552ea2d18c5a57c99e5eabe50898a9deceaa2491c336b626268f58d9acb not found: ID does not exist" Sep 10 13:46:11.496095 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:46:11.496065 2557 scope.go:117] "RemoveContainer" containerID="c9bc7d0ceed89ce9f9cc0ad949f01ea822c607e00482217d447fb53735bfdb0e" Sep 10 13:46:11.496315 ip-10-0-139-174 kubenswrapper[2557]: E0910 13:46:11.496298 2557 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c9bc7d0ceed89ce9f9cc0ad949f01ea822c607e00482217d447fb53735bfdb0e\": container with ID starting with c9bc7d0ceed89ce9f9cc0ad949f01ea822c607e00482217d447fb53735bfdb0e not found: ID does not exist" containerID="c9bc7d0ceed89ce9f9cc0ad949f01ea822c607e00482217d447fb53735bfdb0e" Sep 10 13:46:11.496381 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:46:11.496326 2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c9bc7d0ceed89ce9f9cc0ad949f01ea822c607e00482217d447fb53735bfdb0e"} err="failed to get container status \"c9bc7d0ceed89ce9f9cc0ad949f01ea822c607e00482217d447fb53735bfdb0e\": rpc error: code = NotFound desc = could not find container \"c9bc7d0ceed89ce9f9cc0ad949f01ea822c607e00482217d447fb53735bfdb0e\": container with ID starting with c9bc7d0ceed89ce9f9cc0ad949f01ea822c607e00482217d447fb53735bfdb0e not found: ID does not exist" Sep 10 13:46:11.506045 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:46:11.506025 2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-7657ef-openshift-default-85d55f76f6-gw88p"] Sep 10 13:46:11.525902 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:46:11.525883 2557 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-7657ef-openshift-default-85d55f76f6-gw88p"] Sep 10 13:46:12.220637 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:46:12.220605 2557 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="979718af-bb04-448e-8c47-4c82a9adce63" path="/var/lib/kubelet/pods/979718af-bb04-448e-8c47-4c82a9adce63/volumes" Sep 10 13:46:17.379792 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:46:17.379759 2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-7739cb-79cc8c98c-gn8pb"] Sep 10 13:46:17.380208 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:46:17.380014 2557 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-7739cb-79cc8c98c-gn8pb" podUID="e14e9221-9cf2-44a5-8b2c-b995c364cead" containerName="payload-processing" containerID="cri-o://0fde2177917ece876f2de6d674eadbdfc207b421afcab6f9dd08fb468133b223" gracePeriod=30 Sep 10 13:46:17.402009 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:46:17.401983 2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-7739cb-5b465cd6d6-snwfr"] Sep 10 13:46:17.402244 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:46:17.402197 2557 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-7739cb-5b465cd6d6-snwfr" podUID="c6fa8034-38ba-4d3a-a60a-3355bc14ed94" containerName="payload-pre-processing" containerID="cri-o://31bf4dae536c57381a93f4eb814e6667d414830a9b3f40415ae2ae9b677fb044" gracePeriod=30 Sep 10 13:46:17.516338 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:46:17.516311 2557 generic.go:356] "Generic (PLEG): container finished" podID="e14e9221-9cf2-44a5-8b2c-b995c364cead" containerID="0fde2177917ece876f2de6d674eadbdfc207b421afcab6f9dd08fb468133b223" exitCode=0 Sep 10 13:46:17.516464 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:46:17.516388 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-7739cb-79cc8c98c-gn8pb" event={"ID":"e14e9221-9cf2-44a5-8b2c-b995c364cead","Type":"ContainerDied","Data":"0fde2177917ece876f2de6d674eadbdfc207b421afcab6f9dd08fb468133b223"} Sep 10 13:46:17.684789 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:46:17.684762 2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-7739cb-79cc8c98c-gn8pb" Sep 10 13:46:17.687263 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:46:17.687238 2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-7739cb-5b465cd6d6-snwfr" Sep 10 13:46:17.787283 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:46:17.787248 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-82tzg\" (UniqueName: \"kubernetes.io/projected/c6fa8034-38ba-4d3a-a60a-3355bc14ed94-kube-api-access-82tzg\") pod \"c6fa8034-38ba-4d3a-a60a-3355bc14ed94\" (UID: \"c6fa8034-38ba-4d3a-a60a-3355bc14ed94\") " Sep 10 13:46:17.787487 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:46:17.787405 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c6fa8034-38ba-4d3a-a60a-3355bc14ed94-plugins-config-volume\") pod \"c6fa8034-38ba-4d3a-a60a-3355bc14ed94\" (UID: \"c6fa8034-38ba-4d3a-a60a-3355bc14ed94\") " Sep 10 13:46:17.787566 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:46:17.787491 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e14e9221-9cf2-44a5-8b2c-b995c364cead-plugins-config-volume\") pod \"e14e9221-9cf2-44a5-8b2c-b995c364cead\" (UID: \"e14e9221-9cf2-44a5-8b2c-b995c364cead\") " Sep 10 13:46:17.787566 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:46:17.787541 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hrtxp\" (UniqueName: \"kubernetes.io/projected/e14e9221-9cf2-44a5-8b2c-b995c364cead-kube-api-access-hrtxp\") pod \"e14e9221-9cf2-44a5-8b2c-b995c364cead\" (UID: \"e14e9221-9cf2-44a5-8b2c-b995c364cead\") " Sep 10 13:46:17.788020 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:46:17.787962 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c6fa8034-38ba-4d3a-a60a-3355bc14ed94-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "c6fa8034-38ba-4d3a-a60a-3355bc14ed94" (UID: "c6fa8034-38ba-4d3a-a60a-3355bc14ed94"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 10 13:46:17.788020 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:46:17.787971 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e14e9221-9cf2-44a5-8b2c-b995c364cead-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "e14e9221-9cf2-44a5-8b2c-b995c364cead" (UID: "e14e9221-9cf2-44a5-8b2c-b995c364cead"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 10 13:46:17.789743 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:46:17.789710 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c6fa8034-38ba-4d3a-a60a-3355bc14ed94-kube-api-access-82tzg" (OuterVolumeSpecName: "kube-api-access-82tzg") pod "c6fa8034-38ba-4d3a-a60a-3355bc14ed94" (UID: "c6fa8034-38ba-4d3a-a60a-3355bc14ed94"). InnerVolumeSpecName "kube-api-access-82tzg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 10 13:46:17.790172 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:46:17.790151 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e14e9221-9cf2-44a5-8b2c-b995c364cead-kube-api-access-hrtxp" (OuterVolumeSpecName: "kube-api-access-hrtxp") pod "e14e9221-9cf2-44a5-8b2c-b995c364cead" (UID: "e14e9221-9cf2-44a5-8b2c-b995c364cead"). InnerVolumeSpecName "kube-api-access-hrtxp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 10 13:46:17.888858 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:46:17.888831 2557 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e14e9221-9cf2-44a5-8b2c-b995c364cead-plugins-config-volume\") on node \"ip-10-0-139-174.ec2.internal\" DevicePath \"\"" Sep 10 13:46:17.888858 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:46:17.888856 2557 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hrtxp\" (UniqueName: \"kubernetes.io/projected/e14e9221-9cf2-44a5-8b2c-b995c364cead-kube-api-access-hrtxp\") on node \"ip-10-0-139-174.ec2.internal\" DevicePath \"\"" Sep 10 13:46:17.889043 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:46:17.888872 2557 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-82tzg\" (UniqueName: \"kubernetes.io/projected/c6fa8034-38ba-4d3a-a60a-3355bc14ed94-kube-api-access-82tzg\") on node \"ip-10-0-139-174.ec2.internal\" DevicePath \"\"" Sep 10 13:46:17.889043 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:46:17.888886 2557 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c6fa8034-38ba-4d3a-a60a-3355bc14ed94-plugins-config-volume\") on node \"ip-10-0-139-174.ec2.internal\" DevicePath \"\"" Sep 10 13:46:18.523862 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:46:18.523788 2557 generic.go:356] "Generic (PLEG): container finished" podID="c6fa8034-38ba-4d3a-a60a-3355bc14ed94" containerID="31bf4dae536c57381a93f4eb814e6667d414830a9b3f40415ae2ae9b677fb044" exitCode=0 Sep 10 13:46:18.523862 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:46:18.523833 2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-7739cb-5b465cd6d6-snwfr" Sep 10 13:46:18.524260 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:46:18.523878 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-7739cb-5b465cd6d6-snwfr" event={"ID":"c6fa8034-38ba-4d3a-a60a-3355bc14ed94","Type":"ContainerDied","Data":"31bf4dae536c57381a93f4eb814e6667d414830a9b3f40415ae2ae9b677fb044"} Sep 10 13:46:18.524260 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:46:18.523938 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-7739cb-5b465cd6d6-snwfr" event={"ID":"c6fa8034-38ba-4d3a-a60a-3355bc14ed94","Type":"ContainerDied","Data":"fbf3da115e0d1fc08cc9f48f8ad13730c0bac0ba95bb86a2efdc5dbf61179f20"} Sep 10 13:46:18.524260 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:46:18.523970 2557 scope.go:117] "RemoveContainer" containerID="31bf4dae536c57381a93f4eb814e6667d414830a9b3f40415ae2ae9b677fb044" Sep 10 13:46:18.525568 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:46:18.525481 2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-7739cb-79cc8c98c-gn8pb" Sep 10 13:46:18.525568 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:46:18.525521 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-7739cb-79cc8c98c-gn8pb" event={"ID":"e14e9221-9cf2-44a5-8b2c-b995c364cead","Type":"ContainerDied","Data":"aaa02d262fb9d4f2c562520254b40fa9cc6459d208ec8fcbd8475055465910c1"} Sep 10 13:46:18.535955 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:46:18.535930 2557 scope.go:117] "RemoveContainer" containerID="31bf4dae536c57381a93f4eb814e6667d414830a9b3f40415ae2ae9b677fb044" Sep 10 13:46:18.536332 ip-10-0-139-174 kubenswrapper[2557]: E0910 13:46:18.536315 2557 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"31bf4dae536c57381a93f4eb814e6667d414830a9b3f40415ae2ae9b677fb044\": container with ID starting with 31bf4dae536c57381a93f4eb814e6667d414830a9b3f40415ae2ae9b677fb044 not found: ID does not exist" containerID="31bf4dae536c57381a93f4eb814e6667d414830a9b3f40415ae2ae9b677fb044" Sep 10 13:46:18.536392 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:46:18.536337 2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"31bf4dae536c57381a93f4eb814e6667d414830a9b3f40415ae2ae9b677fb044"} err="failed to get container status \"31bf4dae536c57381a93f4eb814e6667d414830a9b3f40415ae2ae9b677fb044\": rpc error: code = NotFound desc = could not find container \"31bf4dae536c57381a93f4eb814e6667d414830a9b3f40415ae2ae9b677fb044\": container with ID starting with 31bf4dae536c57381a93f4eb814e6667d414830a9b3f40415ae2ae9b677fb044 not found: ID does not exist" Sep 10 13:46:18.536392 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:46:18.536364 2557 scope.go:117] "RemoveContainer" containerID="0fde2177917ece876f2de6d674eadbdfc207b421afcab6f9dd08fb468133b223" Sep 10 13:46:18.546827 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:46:18.546807 2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-7739cb-5b465cd6d6-snwfr"] Sep 10 13:46:18.551113 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:46:18.551092 2557 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-7739cb-5b465cd6d6-snwfr"] Sep 10 13:46:18.563743 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:46:18.563705 2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-7739cb-79cc8c98c-gn8pb"] Sep 10 13:46:18.574159 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:46:18.574142 2557 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-7739cb-79cc8c98c-gn8pb"] Sep 10 13:46:20.219837 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:46:20.219805 2557 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c6fa8034-38ba-4d3a-a60a-3355bc14ed94" path="/var/lib/kubelet/pods/c6fa8034-38ba-4d3a-a60a-3355bc14ed94/volumes" Sep 10 13:46:20.220186 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:46:20.220117 2557 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e14e9221-9cf2-44a5-8b2c-b995c364cead" path="/var/lib/kubelet/pods/e14e9221-9cf2-44a5-8b2c-b995c364cead/volumes" Sep 10 13:46:26.706628 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:46:26.706587 2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-7739cb-openshift-default-6764b957b8-zn8rv"] Sep 10 13:46:26.706986 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:46:26.706826 2557 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w3-7739cb-openshift-default-6764b957b8-zn8rv" podUID="3b257e70-49de-40e6-940f-2306f4a81156" containerName="istio-proxy" containerID="cri-o://f3ace53963b56f13c6cc784048663874a12d9c1583c4495fc4c216e8b44628f2" gracePeriod=30 Sep 10 13:46:29.440083 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:46:29.440052 2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-7878d84c4d-hvwcv"] Sep 10 13:46:29.440571 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:46:29.440224 2557 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/maas-controller-7878d84c4d-hvwcv" podUID="575f105f-5487-444e-8690-0eba38df97c3" containerName="manager" containerID="cri-o://4a54dab47090b536b25a4f5a6fc1de7822fbeed6c9e44e6521f64b09e97cfcb7" gracePeriod=30 Sep 10 13:46:29.591154 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:46:29.591124 2557 generic.go:356] "Generic (PLEG): container finished" podID="575f105f-5487-444e-8690-0eba38df97c3" containerID="4a54dab47090b536b25a4f5a6fc1de7822fbeed6c9e44e6521f64b09e97cfcb7" exitCode=0 Sep 10 13:46:29.591316 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:46:29.591199 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-7878d84c4d-hvwcv" event={"ID":"575f105f-5487-444e-8690-0eba38df97c3","Type":"ContainerDied","Data":"4a54dab47090b536b25a4f5a6fc1de7822fbeed6c9e44e6521f64b09e97cfcb7"} Sep 10 13:46:29.683274 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:46:29.683253 2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-7878d84c4d-hvwcv" Sep 10 13:46:29.688409 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:46:29.688384 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/575f105f-5487-444e-8690-0eba38df97c3-metrics-certs\") pod \"575f105f-5487-444e-8690-0eba38df97c3\" (UID: \"575f105f-5487-444e-8690-0eba38df97c3\") " Sep 10 13:46:29.688541 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:46:29.688508 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-495rf\" (UniqueName: \"kubernetes.io/projected/575f105f-5487-444e-8690-0eba38df97c3-kube-api-access-495rf\") pod \"575f105f-5487-444e-8690-0eba38df97c3\" (UID: \"575f105f-5487-444e-8690-0eba38df97c3\") " Sep 10 13:46:29.688609 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:46:29.688565 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/575f105f-5487-444e-8690-0eba38df97c3-webhook-cert\") pod \"575f105f-5487-444e-8690-0eba38df97c3\" (UID: \"575f105f-5487-444e-8690-0eba38df97c3\") " Sep 10 13:46:29.690628 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:46:29.690569 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/575f105f-5487-444e-8690-0eba38df97c3-kube-api-access-495rf" (OuterVolumeSpecName: "kube-api-access-495rf") pod "575f105f-5487-444e-8690-0eba38df97c3" (UID: "575f105f-5487-444e-8690-0eba38df97c3"). InnerVolumeSpecName "kube-api-access-495rf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 10 13:46:29.690628 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:46:29.690588 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/575f105f-5487-444e-8690-0eba38df97c3-webhook-cert" (OuterVolumeSpecName: "webhook-cert") pod "575f105f-5487-444e-8690-0eba38df97c3" (UID: "575f105f-5487-444e-8690-0eba38df97c3"). InnerVolumeSpecName "webhook-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 10 13:46:29.690742 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:46:29.690630 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/575f105f-5487-444e-8690-0eba38df97c3-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "575f105f-5487-444e-8690-0eba38df97c3" (UID: "575f105f-5487-444e-8690-0eba38df97c3"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 10 13:46:29.789727 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:46:29.789692 2557 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-495rf\" (UniqueName: \"kubernetes.io/projected/575f105f-5487-444e-8690-0eba38df97c3-kube-api-access-495rf\") on node \"ip-10-0-139-174.ec2.internal\" DevicePath \"\"" Sep 10 13:46:29.789727 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:46:29.789723 2557 reconciler_common.go:299] "Volume detached for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/575f105f-5487-444e-8690-0eba38df97c3-webhook-cert\") on node \"ip-10-0-139-174.ec2.internal\" DevicePath \"\"" Sep 10 13:46:29.789727 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:46:29.789732 2557 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/575f105f-5487-444e-8690-0eba38df97c3-metrics-certs\") on node \"ip-10-0-139-174.ec2.internal\" DevicePath \"\"" Sep 10 13:46:30.598376 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:46:30.598294 2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-7878d84c4d-hvwcv" Sep 10 13:46:30.598376 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:46:30.598322 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-7878d84c4d-hvwcv" event={"ID":"575f105f-5487-444e-8690-0eba38df97c3","Type":"ContainerDied","Data":"324042d4a04665d1a2d4c5fa9f88d331e803bd534d31c6a6e5313d35316015da"} Sep 10 13:46:30.598890 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:46:30.598387 2557 scope.go:117] "RemoveContainer" containerID="4a54dab47090b536b25a4f5a6fc1de7822fbeed6c9e44e6521f64b09e97cfcb7" Sep 10 13:46:30.616679 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:46:30.616655 2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-7878d84c4d-hvwcv"] Sep 10 13:46:30.621876 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:46:30.621856 2557 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-controller-7878d84c4d-hvwcv"] Sep 10 13:46:31.953323 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:46:31.953293 2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w3-7739cb-openshift-default-6764b957b8-zn8rv" Sep 10 13:46:32.009648 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:46:32.009570 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3b257e70-49de-40e6-940f-2306f4a81156-workload-certs\") pod \"3b257e70-49de-40e6-940f-2306f4a81156\" (UID: \"3b257e70-49de-40e6-940f-2306f4a81156\") " Sep 10 13:46:32.009648 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:46:32.009615 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3b257e70-49de-40e6-940f-2306f4a81156-credential-socket\") pod \"3b257e70-49de-40e6-940f-2306f4a81156\" (UID: \"3b257e70-49de-40e6-940f-2306f4a81156\") " Sep 10 13:46:32.009854 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:46:32.009664 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3b257e70-49de-40e6-940f-2306f4a81156-istio-token\") pod \"3b257e70-49de-40e6-940f-2306f4a81156\" (UID: \"3b257e70-49de-40e6-940f-2306f4a81156\") " Sep 10 13:46:32.009854 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:46:32.009734 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3b257e70-49de-40e6-940f-2306f4a81156-istiod-ca-cert\") pod \"3b257e70-49de-40e6-940f-2306f4a81156\" (UID: \"3b257e70-49de-40e6-940f-2306f4a81156\") " Sep 10 13:46:32.009854 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:46:32.009756 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3b257e70-49de-40e6-940f-2306f4a81156-workload-socket\") pod \"3b257e70-49de-40e6-940f-2306f4a81156\" (UID: \"3b257e70-49de-40e6-940f-2306f4a81156\") " Sep 10 13:46:32.009854 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:46:32.009783 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3b257e70-49de-40e6-940f-2306f4a81156-istio-podinfo\") pod \"3b257e70-49de-40e6-940f-2306f4a81156\" (UID: \"3b257e70-49de-40e6-940f-2306f4a81156\") " Sep 10 13:46:32.009854 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:46:32.009813 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3b257e70-49de-40e6-940f-2306f4a81156-istio-envoy\") pod \"3b257e70-49de-40e6-940f-2306f4a81156\" (UID: \"3b257e70-49de-40e6-940f-2306f4a81156\") " Sep 10 13:46:32.009854 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:46:32.009847 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3b257e70-49de-40e6-940f-2306f4a81156-istio-data\") pod \"3b257e70-49de-40e6-940f-2306f4a81156\" (UID: \"3b257e70-49de-40e6-940f-2306f4a81156\") " Sep 10 13:46:32.010156 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:46:32.009903 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-tnfhn\" (UniqueName: \"kubernetes.io/projected/3b257e70-49de-40e6-940f-2306f4a81156-kube-api-access-tnfhn\") pod \"3b257e70-49de-40e6-940f-2306f4a81156\" (UID: \"3b257e70-49de-40e6-940f-2306f4a81156\") " Sep 10 13:46:32.010227 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:46:32.009876 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3b257e70-49de-40e6-940f-2306f4a81156-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "3b257e70-49de-40e6-940f-2306f4a81156" (UID: "3b257e70-49de-40e6-940f-2306f4a81156"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 13:46:32.010227 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:46:32.009917 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3b257e70-49de-40e6-940f-2306f4a81156-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "3b257e70-49de-40e6-940f-2306f4a81156" (UID: "3b257e70-49de-40e6-940f-2306f4a81156"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 13:46:32.010227 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:46:32.010070 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3b257e70-49de-40e6-940f-2306f4a81156-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "3b257e70-49de-40e6-940f-2306f4a81156" (UID: "3b257e70-49de-40e6-940f-2306f4a81156"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 10 13:46:32.010227 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:46:32.010080 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3b257e70-49de-40e6-940f-2306f4a81156-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "3b257e70-49de-40e6-940f-2306f4a81156" (UID: "3b257e70-49de-40e6-940f-2306f4a81156"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 13:46:32.010227 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:46:32.010095 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3b257e70-49de-40e6-940f-2306f4a81156-istio-data" (OuterVolumeSpecName: "istio-data") pod "3b257e70-49de-40e6-940f-2306f4a81156" (UID: "3b257e70-49de-40e6-940f-2306f4a81156"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 13:46:32.010668 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:46:32.010474 2557 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3b257e70-49de-40e6-940f-2306f4a81156-istiod-ca-cert\") on node \"ip-10-0-139-174.ec2.internal\" DevicePath \"\"" Sep 10 13:46:32.010668 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:46:32.010492 2557 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3b257e70-49de-40e6-940f-2306f4a81156-workload-socket\") on node \"ip-10-0-139-174.ec2.internal\" DevicePath \"\"" Sep 10 13:46:32.010668 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:46:32.010505 2557 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3b257e70-49de-40e6-940f-2306f4a81156-istio-data\") on node \"ip-10-0-139-174.ec2.internal\" DevicePath \"\"" Sep 10 13:46:32.010668 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:46:32.010518 2557 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3b257e70-49de-40e6-940f-2306f4a81156-workload-certs\") on node \"ip-10-0-139-174.ec2.internal\" DevicePath \"\"" Sep 10 13:46:32.010668 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:46:32.010531 2557 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3b257e70-49de-40e6-940f-2306f4a81156-credential-socket\") on node \"ip-10-0-139-174.ec2.internal\" DevicePath \"\"" Sep 10 13:46:32.011889 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:46:32.011864 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3b257e70-49de-40e6-940f-2306f4a81156-istio-token" (OuterVolumeSpecName: "istio-token") pod "3b257e70-49de-40e6-940f-2306f4a81156" (UID: "3b257e70-49de-40e6-940f-2306f4a81156"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 10 13:46:32.012052 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:46:32.012034 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/3b257e70-49de-40e6-940f-2306f4a81156-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "3b257e70-49de-40e6-940f-2306f4a81156" (UID: "3b257e70-49de-40e6-940f-2306f4a81156"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Sep 10 13:46:32.012202 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:46:32.012184 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3b257e70-49de-40e6-940f-2306f4a81156-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "3b257e70-49de-40e6-940f-2306f4a81156" (UID: "3b257e70-49de-40e6-940f-2306f4a81156"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 13:46:32.012515 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:46:32.012489 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3b257e70-49de-40e6-940f-2306f4a81156-kube-api-access-tnfhn" (OuterVolumeSpecName: "kube-api-access-tnfhn") pod "3b257e70-49de-40e6-940f-2306f4a81156" (UID: "3b257e70-49de-40e6-940f-2306f4a81156"). InnerVolumeSpecName "kube-api-access-tnfhn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 10 13:46:32.111187 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:46:32.111160 2557 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3b257e70-49de-40e6-940f-2306f4a81156-istio-token\") on node \"ip-10-0-139-174.ec2.internal\" DevicePath \"\"" Sep 10 13:46:32.111187 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:46:32.111183 2557 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3b257e70-49de-40e6-940f-2306f4a81156-istio-podinfo\") on node \"ip-10-0-139-174.ec2.internal\" DevicePath \"\"" Sep 10 13:46:32.111187 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:46:32.111191 2557 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3b257e70-49de-40e6-940f-2306f4a81156-istio-envoy\") on node \"ip-10-0-139-174.ec2.internal\" DevicePath \"\"" Sep 10 13:46:32.111387 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:46:32.111201 2557 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tnfhn\" (UniqueName: \"kubernetes.io/projected/3b257e70-49de-40e6-940f-2306f4a81156-kube-api-access-tnfhn\") on node \"ip-10-0-139-174.ec2.internal\" DevicePath \"\"" Sep 10 13:46:32.219251 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:46:32.219221 2557 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="575f105f-5487-444e-8690-0eba38df97c3" path="/var/lib/kubelet/pods/575f105f-5487-444e-8690-0eba38df97c3/volumes" Sep 10 13:46:32.613634 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:46:32.613605 2557 generic.go:356] "Generic (PLEG): container finished" podID="3b257e70-49de-40e6-940f-2306f4a81156" containerID="f3ace53963b56f13c6cc784048663874a12d9c1583c4495fc4c216e8b44628f2" exitCode=0 Sep 10 13:46:32.613761 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:46:32.613650 2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w3-7739cb-openshift-default-6764b957b8-zn8rv" Sep 10 13:46:32.613761 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:46:32.613658 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-7739cb-openshift-default-6764b957b8-zn8rv" event={"ID":"3b257e70-49de-40e6-940f-2306f4a81156","Type":"ContainerDied","Data":"f3ace53963b56f13c6cc784048663874a12d9c1583c4495fc4c216e8b44628f2"} Sep 10 13:46:32.613761 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:46:32.613711 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-7739cb-openshift-default-6764b957b8-zn8rv" event={"ID":"3b257e70-49de-40e6-940f-2306f4a81156","Type":"ContainerDied","Data":"d84805aaa84bd9849eef06220f5bfc6966b1c4bd67adbe43e0f07a4ee7a069d4"} Sep 10 13:46:32.613761 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:46:32.613735 2557 scope.go:117] "RemoveContainer" containerID="f3ace53963b56f13c6cc784048663874a12d9c1583c4495fc4c216e8b44628f2" Sep 10 13:46:32.622817 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:46:32.622802 2557 scope.go:117] "RemoveContainer" containerID="140c2c6a411ab954abe37e465cf3f295d26a01c3b2ae15ca0a7281a252ffcf9e" Sep 10 13:46:32.630672 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:46:32.630650 2557 scope.go:117] "RemoveContainer" containerID="f3ace53963b56f13c6cc784048663874a12d9c1583c4495fc4c216e8b44628f2" Sep 10 13:46:32.630923 ip-10-0-139-174 kubenswrapper[2557]: E0910 13:46:32.630904 2557 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f3ace53963b56f13c6cc784048663874a12d9c1583c4495fc4c216e8b44628f2\": container with ID starting with f3ace53963b56f13c6cc784048663874a12d9c1583c4495fc4c216e8b44628f2 not found: ID does not exist" containerID="f3ace53963b56f13c6cc784048663874a12d9c1583c4495fc4c216e8b44628f2" Sep 10 13:46:32.630996 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:46:32.630931 2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f3ace53963b56f13c6cc784048663874a12d9c1583c4495fc4c216e8b44628f2"} err="failed to get container status \"f3ace53963b56f13c6cc784048663874a12d9c1583c4495fc4c216e8b44628f2\": rpc error: code = NotFound desc = could not find container \"f3ace53963b56f13c6cc784048663874a12d9c1583c4495fc4c216e8b44628f2\": container with ID starting with f3ace53963b56f13c6cc784048663874a12d9c1583c4495fc4c216e8b44628f2 not found: ID does not exist" Sep 10 13:46:32.630996 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:46:32.630947 2557 scope.go:117] "RemoveContainer" containerID="140c2c6a411ab954abe37e465cf3f295d26a01c3b2ae15ca0a7281a252ffcf9e" Sep 10 13:46:32.631162 ip-10-0-139-174 kubenswrapper[2557]: E0910 13:46:32.631149 2557 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"140c2c6a411ab954abe37e465cf3f295d26a01c3b2ae15ca0a7281a252ffcf9e\": container with ID starting with 140c2c6a411ab954abe37e465cf3f295d26a01c3b2ae15ca0a7281a252ffcf9e not found: ID does not exist" containerID="140c2c6a411ab954abe37e465cf3f295d26a01c3b2ae15ca0a7281a252ffcf9e" Sep 10 13:46:32.631205 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:46:32.631166 2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"140c2c6a411ab954abe37e465cf3f295d26a01c3b2ae15ca0a7281a252ffcf9e"} err="failed to get container status \"140c2c6a411ab954abe37e465cf3f295d26a01c3b2ae15ca0a7281a252ffcf9e\": rpc error: code = NotFound desc = could not find container \"140c2c6a411ab954abe37e465cf3f295d26a01c3b2ae15ca0a7281a252ffcf9e\": container with ID starting with 140c2c6a411ab954abe37e465cf3f295d26a01c3b2ae15ca0a7281a252ffcf9e not found: ID does not exist" Sep 10 13:46:32.646317 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:46:32.646286 2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-7739cb-openshift-default-6764b957b8-zn8rv"] Sep 10 13:46:32.666507 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:46:32.666484 2557 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-7739cb-openshift-default-6764b957b8-zn8rv"] Sep 10 13:46:34.219408 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:46:34.219367 2557 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3b257e70-49de-40e6-940f-2306f4a81156" path="/var/lib/kubelet/pods/3b257e70-49de-40e6-940f-2306f4a81156/volumes" Sep 10 13:47:20.312898 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:47:20.312870 2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-92n4q_720acda6-0b15-49ac-ab4e-60de70c4324a/ovn-acl-logging/0.log" Sep 10 13:47:20.322356 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:47:20.322333 2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-92n4q_720acda6-0b15-49ac-ab4e-60de70c4324a/ovn-acl-logging/0.log" Sep 10 13:47:20.551008 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:47:20.550978 2557 scope.go:117] "RemoveContainer" containerID="c6c0cb089164246159b2402f9b049d7d596a0c3f64bd26f7aeae156dae073fbe" Sep 10 13:47:31.527899 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:47:31.527864 2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-7878d84c4d-x5tmf"] Sep 10 13:47:31.528284 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:47:31.528194 2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e14e9221-9cf2-44a5-8b2c-b995c364cead" containerName="payload-processing" Sep 10 13:47:31.528284 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:47:31.528205 2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="e14e9221-9cf2-44a5-8b2c-b995c364cead" containerName="payload-processing" Sep 10 13:47:31.528284 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:47:31.528216 2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="979718af-bb04-448e-8c47-4c82a9adce63" containerName="istio-proxy" Sep 10 13:47:31.528284 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:47:31.528221 2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="979718af-bb04-448e-8c47-4c82a9adce63" containerName="istio-proxy" Sep 10 13:47:31.528284 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:47:31.528229 2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3b257e70-49de-40e6-940f-2306f4a81156" containerName="istio-proxy" Sep 10 13:47:31.528284 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:47:31.528234 2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="3b257e70-49de-40e6-940f-2306f4a81156" containerName="istio-proxy" Sep 10 13:47:31.528284 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:47:31.528239 2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="94de69cd-36ae-4d1e-86c7-2c3b09e8f3f2" containerName="revoke-keys" Sep 10 13:47:31.528284 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:47:31.528245 2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="94de69cd-36ae-4d1e-86c7-2c3b09e8f3f2" containerName="revoke-keys" Sep 10 13:47:31.528284 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:47:31.528258 2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="575f105f-5487-444e-8690-0eba38df97c3" containerName="manager" Sep 10 13:47:31.528284 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:47:31.528263 2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="575f105f-5487-444e-8690-0eba38df97c3" containerName="manager" Sep 10 13:47:31.528284 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:47:31.528273 2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c6fa8034-38ba-4d3a-a60a-3355bc14ed94" containerName="payload-pre-processing" Sep 10 13:47:31.528284 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:47:31.528277 2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="c6fa8034-38ba-4d3a-a60a-3355bc14ed94" containerName="payload-pre-processing" Sep 10 13:47:31.528284 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:47:31.528283 2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3b257e70-49de-40e6-940f-2306f4a81156" containerName="istio-proxy" Sep 10 13:47:31.528284 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:47:31.528288 2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="3b257e70-49de-40e6-940f-2306f4a81156" containerName="istio-proxy" Sep 10 13:47:31.528807 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:47:31.528336 2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="3b257e70-49de-40e6-940f-2306f4a81156" containerName="istio-proxy" Sep 10 13:47:31.528807 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:47:31.528342 2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="e14e9221-9cf2-44a5-8b2c-b995c364cead" containerName="payload-processing" Sep 10 13:47:31.528807 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:47:31.528348 2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="94de69cd-36ae-4d1e-86c7-2c3b09e8f3f2" containerName="revoke-keys" Sep 10 13:47:31.528807 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:47:31.528356 2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="575f105f-5487-444e-8690-0eba38df97c3" containerName="manager" Sep 10 13:47:31.528807 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:47:31.528361 2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="c6fa8034-38ba-4d3a-a60a-3355bc14ed94" containerName="payload-pre-processing" Sep 10 13:47:31.528807 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:47:31.528367 2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="979718af-bb04-448e-8c47-4c82a9adce63" containerName="istio-proxy" Sep 10 13:47:31.528807 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:47:31.528371 2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="3b257e70-49de-40e6-940f-2306f4a81156" containerName="istio-proxy" Sep 10 13:47:31.528807 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:47:31.528375 2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="979718af-bb04-448e-8c47-4c82a9adce63" containerName="istio-proxy" Sep 10 13:47:31.531247 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:47:31.531229 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-7878d84c4d-x5tmf" Sep 10 13:47:31.538743 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:47:31.538715 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\"" Sep 10 13:47:31.538858 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:47:31.538729 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-jc6jm\"" Sep 10 13:47:31.542593 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:47:31.542579 2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\"" Sep 10 13:47:31.542885 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:47:31.542871 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-metrics-tls\"" Sep 10 13:47:31.551154 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:47:31.551127 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-7878d84c4d-x5tmf"] Sep 10 13:47:31.583125 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:47:31.583097 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xgp96\" (UniqueName: \"kubernetes.io/projected/45b9a3f9-8720-40d2-afea-c7f99946bfbf-kube-api-access-xgp96\") pod \"maas-controller-7878d84c4d-x5tmf\" (UID: \"45b9a3f9-8720-40d2-afea-c7f99946bfbf\") " pod="opendatahub/maas-controller-7878d84c4d-x5tmf" Sep 10 13:47:31.583125 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:47:31.583127 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/45b9a3f9-8720-40d2-afea-c7f99946bfbf-metrics-certs\") pod \"maas-controller-7878d84c4d-x5tmf\" (UID: \"45b9a3f9-8720-40d2-afea-c7f99946bfbf\") " pod="opendatahub/maas-controller-7878d84c4d-x5tmf" Sep 10 13:47:31.583295 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:47:31.583221 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/45b9a3f9-8720-40d2-afea-c7f99946bfbf-webhook-cert\") pod \"maas-controller-7878d84c4d-x5tmf\" (UID: \"45b9a3f9-8720-40d2-afea-c7f99946bfbf\") " pod="opendatahub/maas-controller-7878d84c4d-x5tmf" Sep 10 13:47:31.683648 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:47:31.683615 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xgp96\" (UniqueName: \"kubernetes.io/projected/45b9a3f9-8720-40d2-afea-c7f99946bfbf-kube-api-access-xgp96\") pod \"maas-controller-7878d84c4d-x5tmf\" (UID: \"45b9a3f9-8720-40d2-afea-c7f99946bfbf\") " pod="opendatahub/maas-controller-7878d84c4d-x5tmf" Sep 10 13:47:31.683795 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:47:31.683653 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/45b9a3f9-8720-40d2-afea-c7f99946bfbf-metrics-certs\") pod \"maas-controller-7878d84c4d-x5tmf\" (UID: \"45b9a3f9-8720-40d2-afea-c7f99946bfbf\") " pod="opendatahub/maas-controller-7878d84c4d-x5tmf" Sep 10 13:47:31.683795 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:47:31.683723 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/45b9a3f9-8720-40d2-afea-c7f99946bfbf-webhook-cert\") pod \"maas-controller-7878d84c4d-x5tmf\" (UID: \"45b9a3f9-8720-40d2-afea-c7f99946bfbf\") " pod="opendatahub/maas-controller-7878d84c4d-x5tmf" Sep 10 13:47:31.686150 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:47:31.686125 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/45b9a3f9-8720-40d2-afea-c7f99946bfbf-metrics-certs\") pod \"maas-controller-7878d84c4d-x5tmf\" (UID: \"45b9a3f9-8720-40d2-afea-c7f99946bfbf\") " pod="opendatahub/maas-controller-7878d84c4d-x5tmf" Sep 10 13:47:31.686244 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:47:31.686169 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/45b9a3f9-8720-40d2-afea-c7f99946bfbf-webhook-cert\") pod \"maas-controller-7878d84c4d-x5tmf\" (UID: \"45b9a3f9-8720-40d2-afea-c7f99946bfbf\") " pod="opendatahub/maas-controller-7878d84c4d-x5tmf" Sep 10 13:47:31.692323 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:47:31.692304 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xgp96\" (UniqueName: \"kubernetes.io/projected/45b9a3f9-8720-40d2-afea-c7f99946bfbf-kube-api-access-xgp96\") pod \"maas-controller-7878d84c4d-x5tmf\" (UID: \"45b9a3f9-8720-40d2-afea-c7f99946bfbf\") " pod="opendatahub/maas-controller-7878d84c4d-x5tmf" Sep 10 13:47:31.841527 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:47:31.841457 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-7878d84c4d-x5tmf" Sep 10 13:47:32.117099 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:47:32.117070 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-7878d84c4d-x5tmf"] Sep 10 13:47:32.119086 ip-10-0-139-174 kubenswrapper[2557]: W0910 13:47:32.119062 2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod45b9a3f9_8720_40d2_afea_c7f99946bfbf.slice/crio-26221a77a756e25bbd25814184cf5f97716ff1cc23db19b3b1a1cd943d5d86b7 WatchSource:0}: Error finding container 26221a77a756e25bbd25814184cf5f97716ff1cc23db19b3b1a1cd943d5d86b7: Status 404 returned error can't find the container with id 26221a77a756e25bbd25814184cf5f97716ff1cc23db19b3b1a1cd943d5d86b7 Sep 10 13:47:32.120260 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:47:32.120243 2557 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Sep 10 13:47:32.959082 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:47:32.959041 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-7878d84c4d-x5tmf" event={"ID":"45b9a3f9-8720-40d2-afea-c7f99946bfbf","Type":"ContainerStarted","Data":"55296948c8b08e42ec3fbf5e99e2fc2065f3894a7899cbba9047ba5bf73d80f1"} Sep 10 13:47:32.959528 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:47:32.959119 2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-7878d84c4d-x5tmf" Sep 10 13:47:32.959528 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:47:32.959144 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-7878d84c4d-x5tmf" event={"ID":"45b9a3f9-8720-40d2-afea-c7f99946bfbf","Type":"ContainerStarted","Data":"26221a77a756e25bbd25814184cf5f97716ff1cc23db19b3b1a1cd943d5d86b7"} Sep 10 13:47:32.989507 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:47:32.989458 2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-controller-7878d84c4d-x5tmf" podStartSLOduration=1.477364076 podStartE2EDuration="1.989442344s" totalImagesPullingTime="512.078268ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 13:47:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 13:47:32.987981587 +0000 UTC m=+1813.294976000" watchObservedRunningTime="2026-09-10 13:47:32.989442344 +0000 UTC m=+1813.296436750" Sep 10 13:47:43.966798 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:47:43.966759 2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-7878d84c4d-x5tmf" Sep 10 13:47:53.450960 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:47:53.450920 2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-7878d84c4d-x5tmf"] Sep 10 13:47:53.451613 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:47:53.451159 2557 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/maas-controller-7878d84c4d-x5tmf" podUID="45b9a3f9-8720-40d2-afea-c7f99946bfbf" containerName="manager" containerID="cri-o://55296948c8b08e42ec3fbf5e99e2fc2065f3894a7899cbba9047ba5bf73d80f1" gracePeriod=30 Sep 10 13:47:53.708664 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:47:53.708592 2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-7878d84c4d-x5tmf" Sep 10 13:47:53.761055 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:47:53.760193 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xgp96\" (UniqueName: \"kubernetes.io/projected/45b9a3f9-8720-40d2-afea-c7f99946bfbf-kube-api-access-xgp96\") pod \"45b9a3f9-8720-40d2-afea-c7f99946bfbf\" (UID: \"45b9a3f9-8720-40d2-afea-c7f99946bfbf\") " Sep 10 13:47:53.761055 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:47:53.760321 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/45b9a3f9-8720-40d2-afea-c7f99946bfbf-webhook-cert\") pod \"45b9a3f9-8720-40d2-afea-c7f99946bfbf\" (UID: \"45b9a3f9-8720-40d2-afea-c7f99946bfbf\") " Sep 10 13:47:53.761055 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:47:53.760366 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/45b9a3f9-8720-40d2-afea-c7f99946bfbf-metrics-certs\") pod \"45b9a3f9-8720-40d2-afea-c7f99946bfbf\" (UID: \"45b9a3f9-8720-40d2-afea-c7f99946bfbf\") " Sep 10 13:47:53.763390 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:47:53.763345 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/45b9a3f9-8720-40d2-afea-c7f99946bfbf-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "45b9a3f9-8720-40d2-afea-c7f99946bfbf" (UID: "45b9a3f9-8720-40d2-afea-c7f99946bfbf"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 10 13:47:53.766192 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:47:53.766090 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/45b9a3f9-8720-40d2-afea-c7f99946bfbf-kube-api-access-xgp96" (OuterVolumeSpecName: "kube-api-access-xgp96") pod "45b9a3f9-8720-40d2-afea-c7f99946bfbf" (UID: "45b9a3f9-8720-40d2-afea-c7f99946bfbf"). InnerVolumeSpecName "kube-api-access-xgp96". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 10 13:47:53.766192 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:47:53.766164 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/45b9a3f9-8720-40d2-afea-c7f99946bfbf-webhook-cert" (OuterVolumeSpecName: "webhook-cert") pod "45b9a3f9-8720-40d2-afea-c7f99946bfbf" (UID: "45b9a3f9-8720-40d2-afea-c7f99946bfbf"). InnerVolumeSpecName "webhook-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 10 13:47:53.861628 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:47:53.861591 2557 reconciler_common.go:299] "Volume detached for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/45b9a3f9-8720-40d2-afea-c7f99946bfbf-webhook-cert\") on node \"ip-10-0-139-174.ec2.internal\" DevicePath \"\"" Sep 10 13:47:53.861628 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:47:53.861617 2557 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/45b9a3f9-8720-40d2-afea-c7f99946bfbf-metrics-certs\") on node \"ip-10-0-139-174.ec2.internal\" DevicePath \"\"" Sep 10 13:47:53.861628 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:47:53.861625 2557 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xgp96\" (UniqueName: \"kubernetes.io/projected/45b9a3f9-8720-40d2-afea-c7f99946bfbf-kube-api-access-xgp96\") on node \"ip-10-0-139-174.ec2.internal\" DevicePath \"\"" Sep 10 13:47:54.080583 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:47:54.080504 2557 generic.go:356] "Generic (PLEG): container finished" podID="45b9a3f9-8720-40d2-afea-c7f99946bfbf" containerID="55296948c8b08e42ec3fbf5e99e2fc2065f3894a7899cbba9047ba5bf73d80f1" exitCode=0 Sep 10 13:47:54.080583 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:47:54.080538 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-7878d84c4d-x5tmf" event={"ID":"45b9a3f9-8720-40d2-afea-c7f99946bfbf","Type":"ContainerDied","Data":"55296948c8b08e42ec3fbf5e99e2fc2065f3894a7899cbba9047ba5bf73d80f1"} Sep 10 13:47:54.080793 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:47:54.080594 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-7878d84c4d-x5tmf" event={"ID":"45b9a3f9-8720-40d2-afea-c7f99946bfbf","Type":"ContainerDied","Data":"26221a77a756e25bbd25814184cf5f97716ff1cc23db19b3b1a1cd943d5d86b7"} Sep 10 13:47:54.080793 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:47:54.080618 2557 scope.go:117] "RemoveContainer" containerID="55296948c8b08e42ec3fbf5e99e2fc2065f3894a7899cbba9047ba5bf73d80f1" Sep 10 13:47:54.080793 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:47:54.080549 2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-7878d84c4d-x5tmf" Sep 10 13:47:54.094197 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:47:54.094178 2557 scope.go:117] "RemoveContainer" containerID="55296948c8b08e42ec3fbf5e99e2fc2065f3894a7899cbba9047ba5bf73d80f1" Sep 10 13:47:54.094503 ip-10-0-139-174 kubenswrapper[2557]: E0910 13:47:54.094486 2557 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"55296948c8b08e42ec3fbf5e99e2fc2065f3894a7899cbba9047ba5bf73d80f1\": container with ID starting with 55296948c8b08e42ec3fbf5e99e2fc2065f3894a7899cbba9047ba5bf73d80f1 not found: ID does not exist" containerID="55296948c8b08e42ec3fbf5e99e2fc2065f3894a7899cbba9047ba5bf73d80f1" Sep 10 13:47:54.094553 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:47:54.094511 2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"55296948c8b08e42ec3fbf5e99e2fc2065f3894a7899cbba9047ba5bf73d80f1"} err="failed to get container status \"55296948c8b08e42ec3fbf5e99e2fc2065f3894a7899cbba9047ba5bf73d80f1\": rpc error: code = NotFound desc = could not find container \"55296948c8b08e42ec3fbf5e99e2fc2065f3894a7899cbba9047ba5bf73d80f1\": container with ID starting with 55296948c8b08e42ec3fbf5e99e2fc2065f3894a7899cbba9047ba5bf73d80f1 not found: ID does not exist" Sep 10 13:47:54.107755 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:47:54.107720 2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-7878d84c4d-x5tmf"] Sep 10 13:47:54.116378 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:47:54.116347 2557 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-controller-7878d84c4d-x5tmf"] Sep 10 13:47:54.220947 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:47:54.220922 2557 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="45b9a3f9-8720-40d2-afea-c7f99946bfbf" path="/var/lib/kubelet/pods/45b9a3f9-8720-40d2-afea-c7f99946bfbf/volumes" Sep 10 13:47:54.986033 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:47:54.985998 2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-7878d84c4d-wmfmn"] Sep 10 13:47:54.986536 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:47:54.986518 2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="45b9a3f9-8720-40d2-afea-c7f99946bfbf" containerName="manager" Sep 10 13:47:54.986626 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:47:54.986538 2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="45b9a3f9-8720-40d2-afea-c7f99946bfbf" containerName="manager" Sep 10 13:47:54.986626 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:47:54.986554 2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="979718af-bb04-448e-8c47-4c82a9adce63" containerName="istio-proxy" Sep 10 13:47:54.986626 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:47:54.986565 2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="979718af-bb04-448e-8c47-4c82a9adce63" containerName="istio-proxy" Sep 10 13:47:54.986778 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:47:54.986654 2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="45b9a3f9-8720-40d2-afea-c7f99946bfbf" containerName="manager" Sep 10 13:47:54.990981 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:47:54.990961 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-7878d84c4d-wmfmn" Sep 10 13:47:54.993732 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:47:54.993708 2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\"" Sep 10 13:47:54.993858 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:47:54.993707 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-jc6jm\"" Sep 10 13:47:54.993858 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:47:54.993709 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\"" Sep 10 13:47:54.993858 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:47:54.993710 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-metrics-tls\"" Sep 10 13:47:55.006383 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:47:55.006361 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-7878d84c4d-wmfmn"] Sep 10 13:47:55.073101 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:47:55.073074 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bstsl\" (UniqueName: \"kubernetes.io/projected/83246f1d-5fd7-4ae8-95b2-b94d9660b31a-kube-api-access-bstsl\") pod \"maas-controller-7878d84c4d-wmfmn\" (UID: \"83246f1d-5fd7-4ae8-95b2-b94d9660b31a\") " pod="opendatahub/maas-controller-7878d84c4d-wmfmn" Sep 10 13:47:55.073207 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:47:55.073107 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/83246f1d-5fd7-4ae8-95b2-b94d9660b31a-webhook-cert\") pod \"maas-controller-7878d84c4d-wmfmn\" (UID: \"83246f1d-5fd7-4ae8-95b2-b94d9660b31a\") " pod="opendatahub/maas-controller-7878d84c4d-wmfmn" Sep 10 13:47:55.073207 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:47:55.073129 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/83246f1d-5fd7-4ae8-95b2-b94d9660b31a-metrics-certs\") pod \"maas-controller-7878d84c4d-wmfmn\" (UID: \"83246f1d-5fd7-4ae8-95b2-b94d9660b31a\") " pod="opendatahub/maas-controller-7878d84c4d-wmfmn" Sep 10 13:47:55.173774 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:47:55.173750 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bstsl\" (UniqueName: \"kubernetes.io/projected/83246f1d-5fd7-4ae8-95b2-b94d9660b31a-kube-api-access-bstsl\") pod \"maas-controller-7878d84c4d-wmfmn\" (UID: \"83246f1d-5fd7-4ae8-95b2-b94d9660b31a\") " pod="opendatahub/maas-controller-7878d84c4d-wmfmn" Sep 10 13:47:55.173925 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:47:55.173784 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/83246f1d-5fd7-4ae8-95b2-b94d9660b31a-webhook-cert\") pod \"maas-controller-7878d84c4d-wmfmn\" (UID: \"83246f1d-5fd7-4ae8-95b2-b94d9660b31a\") " pod="opendatahub/maas-controller-7878d84c4d-wmfmn" Sep 10 13:47:55.173925 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:47:55.173805 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/83246f1d-5fd7-4ae8-95b2-b94d9660b31a-metrics-certs\") pod \"maas-controller-7878d84c4d-wmfmn\" (UID: \"83246f1d-5fd7-4ae8-95b2-b94d9660b31a\") " pod="opendatahub/maas-controller-7878d84c4d-wmfmn" Sep 10 13:47:55.176400 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:47:55.176371 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/83246f1d-5fd7-4ae8-95b2-b94d9660b31a-metrics-certs\") pod \"maas-controller-7878d84c4d-wmfmn\" (UID: \"83246f1d-5fd7-4ae8-95b2-b94d9660b31a\") " pod="opendatahub/maas-controller-7878d84c4d-wmfmn" Sep 10 13:47:55.176515 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:47:55.176371 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/83246f1d-5fd7-4ae8-95b2-b94d9660b31a-webhook-cert\") pod \"maas-controller-7878d84c4d-wmfmn\" (UID: \"83246f1d-5fd7-4ae8-95b2-b94d9660b31a\") " pod="opendatahub/maas-controller-7878d84c4d-wmfmn" Sep 10 13:47:55.184650 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:47:55.184631 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bstsl\" (UniqueName: \"kubernetes.io/projected/83246f1d-5fd7-4ae8-95b2-b94d9660b31a-kube-api-access-bstsl\") pod \"maas-controller-7878d84c4d-wmfmn\" (UID: \"83246f1d-5fd7-4ae8-95b2-b94d9660b31a\") " pod="opendatahub/maas-controller-7878d84c4d-wmfmn" Sep 10 13:47:55.300717 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:47:55.300658 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-7878d84c4d-wmfmn" Sep 10 13:47:55.574460 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:47:55.574438 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-7878d84c4d-wmfmn"] Sep 10 13:47:55.576197 ip-10-0-139-174 kubenswrapper[2557]: W0910 13:47:55.576161 2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod83246f1d_5fd7_4ae8_95b2_b94d9660b31a.slice/crio-c91342cd3e907e4125a456aaa20dbd90a820bf4277ca70448035ed6ce28b93cf WatchSource:0}: Error finding container c91342cd3e907e4125a456aaa20dbd90a820bf4277ca70448035ed6ce28b93cf: Status 404 returned error can't find the container with id c91342cd3e907e4125a456aaa20dbd90a820bf4277ca70448035ed6ce28b93cf Sep 10 13:47:56.094619 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:47:56.094580 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-7878d84c4d-wmfmn" event={"ID":"83246f1d-5fd7-4ae8-95b2-b94d9660b31a","Type":"ContainerStarted","Data":"c91342cd3e907e4125a456aaa20dbd90a820bf4277ca70448035ed6ce28b93cf"} Sep 10 13:47:57.100904 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:47:57.100867 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-7878d84c4d-wmfmn" event={"ID":"83246f1d-5fd7-4ae8-95b2-b94d9660b31a","Type":"ContainerStarted","Data":"818dd1f7b49533cd4026811f8660cb9e985338361ff5929a3b7baedf659392d0"} Sep 10 13:47:57.101276 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:47:57.100920 2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-7878d84c4d-wmfmn" Sep 10 13:48:08.109202 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:48:08.109169 2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-7878d84c4d-wmfmn" Sep 10 13:48:08.133260 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:48:08.133210 2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-controller-7878d84c4d-wmfmn" podStartSLOduration=13.555554447 podStartE2EDuration="14.133195191s" totalImagesPullingTime="577.640744ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 13:47:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 13:47:57.133268613 +0000 UTC m=+1837.440263026" watchObservedRunningTime="2026-09-10 13:48:08.133195191 +0000 UTC m=+1848.440189604" Sep 10 13:48:20.579263 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:48:20.579232 2557 scope.go:117] "RemoveContainer" containerID="c501ba5ea4c4fb70a601ca75da4db8513f4451f6eb0ce1139b323a5fbc4e5ca5" Sep 10 13:48:20.587937 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:48:20.587920 2557 scope.go:117] "RemoveContainer" containerID="04c9e6c2e58e0b8fb6ae0bbad11a4dd4aa208b6c9f287ed6d8dfef7008512a07" Sep 10 13:50:20.622453 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:50:20.622351 2557 scope.go:117] "RemoveContainer" containerID="3e0725ebff17c25c3d5e40e06dbda967690e178efc401e957740dc3754e81dc5" Sep 10 13:51:20.643182 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:51:20.643150 2557 scope.go:117] "RemoveContainer" containerID="ee4d9c10ad26efaff207cd37a4072f0baeb28f5ae539406966adce309de14ce7" Sep 10 13:51:20.651547 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:51:20.651531 2557 scope.go:117] "RemoveContainer" containerID="35fdb6de4b42903ec326366d0582c5931b3b3e55f37add1710a1984ff669d88b" Sep 10 13:52:11.135918 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:52:11.135827 2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-85b6f7bf49-rlbfc"] Sep 10 13:52:11.139600 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:52:11.139579 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-85b6f7bf49-rlbfc" Sep 10 13:52:11.142821 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:52:11.142796 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-serving-cert\"" Sep 10 13:52:11.142939 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:52:11.142796 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\"" Sep 10 13:52:11.142939 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:52:11.142854 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-5fn9n\"" Sep 10 13:52:11.142939 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:52:11.142824 2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Sep 10 13:52:11.142939 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:52:11.142802 2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Sep 10 13:52:11.152583 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:52:11.152556 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-85b6f7bf49-rlbfc"] Sep 10 13:52:11.236761 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:52:11.236720 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9b8c43f1-3245-48ff-ab59-3e38e74ab93f-openshift-service-ca\") pod \"maas-api-85b6f7bf49-rlbfc\" (UID: \"9b8c43f1-3245-48ff-ab59-3e38e74ab93f\") " pod="odh-ai-gateway-infra/maas-api-85b6f7bf49-rlbfc" Sep 10 13:52:11.236761 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:52:11.236765 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9b8c43f1-3245-48ff-ab59-3e38e74ab93f-maas-api-tls\") pod \"maas-api-85b6f7bf49-rlbfc\" (UID: \"9b8c43f1-3245-48ff-ab59-3e38e74ab93f\") " pod="odh-ai-gateway-infra/maas-api-85b6f7bf49-rlbfc" Sep 10 13:52:11.237039 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:52:11.236821 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jql82\" (UniqueName: \"kubernetes.io/projected/9b8c43f1-3245-48ff-ab59-3e38e74ab93f-kube-api-access-jql82\") pod \"maas-api-85b6f7bf49-rlbfc\" (UID: \"9b8c43f1-3245-48ff-ab59-3e38e74ab93f\") " pod="odh-ai-gateway-infra/maas-api-85b6f7bf49-rlbfc" Sep 10 13:52:11.237039 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:52:11.236879 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9b8c43f1-3245-48ff-ab59-3e38e74ab93f-metrics-certs\") pod \"maas-api-85b6f7bf49-rlbfc\" (UID: \"9b8c43f1-3245-48ff-ab59-3e38e74ab93f\") " pod="odh-ai-gateway-infra/maas-api-85b6f7bf49-rlbfc" Sep 10 13:52:11.337935 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:52:11.337869 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9b8c43f1-3245-48ff-ab59-3e38e74ab93f-openshift-service-ca\") pod \"maas-api-85b6f7bf49-rlbfc\" (UID: \"9b8c43f1-3245-48ff-ab59-3e38e74ab93f\") " pod="odh-ai-gateway-infra/maas-api-85b6f7bf49-rlbfc" Sep 10 13:52:11.337935 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:52:11.337943 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9b8c43f1-3245-48ff-ab59-3e38e74ab93f-maas-api-tls\") pod \"maas-api-85b6f7bf49-rlbfc\" (UID: \"9b8c43f1-3245-48ff-ab59-3e38e74ab93f\") " pod="odh-ai-gateway-infra/maas-api-85b6f7bf49-rlbfc" Sep 10 13:52:11.338205 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:52:11.337982 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jql82\" (UniqueName: \"kubernetes.io/projected/9b8c43f1-3245-48ff-ab59-3e38e74ab93f-kube-api-access-jql82\") pod \"maas-api-85b6f7bf49-rlbfc\" (UID: \"9b8c43f1-3245-48ff-ab59-3e38e74ab93f\") " pod="odh-ai-gateway-infra/maas-api-85b6f7bf49-rlbfc" Sep 10 13:52:11.338205 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:52:11.338002 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9b8c43f1-3245-48ff-ab59-3e38e74ab93f-metrics-certs\") pod \"maas-api-85b6f7bf49-rlbfc\" (UID: \"9b8c43f1-3245-48ff-ab59-3e38e74ab93f\") " pod="odh-ai-gateway-infra/maas-api-85b6f7bf49-rlbfc" Sep 10 13:52:11.338631 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:52:11.338602 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9b8c43f1-3245-48ff-ab59-3e38e74ab93f-openshift-service-ca\") pod \"maas-api-85b6f7bf49-rlbfc\" (UID: \"9b8c43f1-3245-48ff-ab59-3e38e74ab93f\") " pod="odh-ai-gateway-infra/maas-api-85b6f7bf49-rlbfc" Sep 10 13:52:11.340960 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:52:11.340933 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9b8c43f1-3245-48ff-ab59-3e38e74ab93f-maas-api-tls\") pod \"maas-api-85b6f7bf49-rlbfc\" (UID: \"9b8c43f1-3245-48ff-ab59-3e38e74ab93f\") " pod="odh-ai-gateway-infra/maas-api-85b6f7bf49-rlbfc" Sep 10 13:52:11.341092 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:52:11.340957 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9b8c43f1-3245-48ff-ab59-3e38e74ab93f-metrics-certs\") pod \"maas-api-85b6f7bf49-rlbfc\" (UID: \"9b8c43f1-3245-48ff-ab59-3e38e74ab93f\") " pod="odh-ai-gateway-infra/maas-api-85b6f7bf49-rlbfc" Sep 10 13:52:11.346864 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:52:11.346840 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jql82\" (UniqueName: \"kubernetes.io/projected/9b8c43f1-3245-48ff-ab59-3e38e74ab93f-kube-api-access-jql82\") pod \"maas-api-85b6f7bf49-rlbfc\" (UID: \"9b8c43f1-3245-48ff-ab59-3e38e74ab93f\") " pod="odh-ai-gateway-infra/maas-api-85b6f7bf49-rlbfc" Sep 10 13:52:11.451783 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:52:11.451733 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-85b6f7bf49-rlbfc" Sep 10 13:52:11.739716 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:52:11.739690 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-85b6f7bf49-rlbfc"] Sep 10 13:52:11.741659 ip-10-0-139-174 kubenswrapper[2557]: W0910 13:52:11.741634 2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9b8c43f1_3245_48ff_ab59_3e38e74ab93f.slice/crio-fa77af55adf74b529026c7303efdcc8809b16aa3fddfd56490cad1469b770121 WatchSource:0}: Error finding container fa77af55adf74b529026c7303efdcc8809b16aa3fddfd56490cad1469b770121: Status 404 returned error can't find the container with id fa77af55adf74b529026c7303efdcc8809b16aa3fddfd56490cad1469b770121 Sep 10 13:52:12.491684 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:52:12.491644 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-85b6f7bf49-rlbfc" event={"ID":"9b8c43f1-3245-48ff-ab59-3e38e74ab93f","Type":"ContainerStarted","Data":"fa77af55adf74b529026c7303efdcc8809b16aa3fddfd56490cad1469b770121"} Sep 10 13:52:14.512277 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:52:14.512241 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-85b6f7bf49-rlbfc" event={"ID":"9b8c43f1-3245-48ff-ab59-3e38e74ab93f","Type":"ContainerStarted","Data":"62f2ac5b38653d7dd228b5437aee9309732ed131c87360d3336976968716f663"} Sep 10 13:52:14.512686 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:52:14.512304 2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-85b6f7bf49-rlbfc" Sep 10 13:52:14.535167 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:52:14.535121 2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-85b6f7bf49-rlbfc" podStartSLOduration=1.668862397 podStartE2EDuration="3.535107077s" totalImagesPullingTime="1.86624468s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 13:52:11 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 13:52:14.53268695 +0000 UTC m=+2094.839681373" watchObservedRunningTime="2026-09-10 13:52:14.535107077 +0000 UTC m=+2094.842101489" Sep 10 13:52:19.170224 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:52:19.170188 2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-85b6f7bf49-rlbfc"] Sep 10 13:52:19.170699 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:52:19.170378 2557 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-85b6f7bf49-rlbfc" podUID="9b8c43f1-3245-48ff-ab59-3e38e74ab93f" containerName="maas-api" containerID="cri-o://62f2ac5b38653d7dd228b5437aee9309732ed131c87360d3336976968716f663" gracePeriod=30 Sep 10 13:52:19.176093 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:52:19.176071 2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-85b6f7bf49-rlbfc" Sep 10 13:52:19.416047 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:52:19.416024 2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-85b6f7bf49-rlbfc" Sep 10 13:52:19.511848 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:52:19.511818 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9b8c43f1-3245-48ff-ab59-3e38e74ab93f-openshift-service-ca\") pod \"9b8c43f1-3245-48ff-ab59-3e38e74ab93f\" (UID: \"9b8c43f1-3245-48ff-ab59-3e38e74ab93f\") " Sep 10 13:52:19.511997 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:52:19.511919 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jql82\" (UniqueName: \"kubernetes.io/projected/9b8c43f1-3245-48ff-ab59-3e38e74ab93f-kube-api-access-jql82\") pod \"9b8c43f1-3245-48ff-ab59-3e38e74ab93f\" (UID: \"9b8c43f1-3245-48ff-ab59-3e38e74ab93f\") " Sep 10 13:52:19.511997 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:52:19.511951 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9b8c43f1-3245-48ff-ab59-3e38e74ab93f-maas-api-tls\") pod \"9b8c43f1-3245-48ff-ab59-3e38e74ab93f\" (UID: \"9b8c43f1-3245-48ff-ab59-3e38e74ab93f\") " Sep 10 13:52:19.511997 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:52:19.511985 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9b8c43f1-3245-48ff-ab59-3e38e74ab93f-metrics-certs\") pod \"9b8c43f1-3245-48ff-ab59-3e38e74ab93f\" (UID: \"9b8c43f1-3245-48ff-ab59-3e38e74ab93f\") " Sep 10 13:52:19.512230 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:52:19.512198 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9b8c43f1-3245-48ff-ab59-3e38e74ab93f-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "9b8c43f1-3245-48ff-ab59-3e38e74ab93f" (UID: "9b8c43f1-3245-48ff-ab59-3e38e74ab93f"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 10 13:52:19.514174 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:52:19.514143 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9b8c43f1-3245-48ff-ab59-3e38e74ab93f-kube-api-access-jql82" (OuterVolumeSpecName: "kube-api-access-jql82") pod "9b8c43f1-3245-48ff-ab59-3e38e74ab93f" (UID: "9b8c43f1-3245-48ff-ab59-3e38e74ab93f"). InnerVolumeSpecName "kube-api-access-jql82". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 10 13:52:19.514282 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:52:19.514168 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9b8c43f1-3245-48ff-ab59-3e38e74ab93f-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "9b8c43f1-3245-48ff-ab59-3e38e74ab93f" (UID: "9b8c43f1-3245-48ff-ab59-3e38e74ab93f"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 10 13:52:19.514282 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:52:19.514240 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9b8c43f1-3245-48ff-ab59-3e38e74ab93f-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "9b8c43f1-3245-48ff-ab59-3e38e74ab93f" (UID: "9b8c43f1-3245-48ff-ab59-3e38e74ab93f"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 10 13:52:19.539956 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:52:19.539926 2557 generic.go:356] "Generic (PLEG): container finished" podID="9b8c43f1-3245-48ff-ab59-3e38e74ab93f" containerID="62f2ac5b38653d7dd228b5437aee9309732ed131c87360d3336976968716f663" exitCode=0 Sep 10 13:52:19.540048 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:52:19.539975 2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-85b6f7bf49-rlbfc" Sep 10 13:52:19.540048 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:52:19.539996 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-85b6f7bf49-rlbfc" event={"ID":"9b8c43f1-3245-48ff-ab59-3e38e74ab93f","Type":"ContainerDied","Data":"62f2ac5b38653d7dd228b5437aee9309732ed131c87360d3336976968716f663"} Sep 10 13:52:19.540048 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:52:19.540042 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-85b6f7bf49-rlbfc" event={"ID":"9b8c43f1-3245-48ff-ab59-3e38e74ab93f","Type":"ContainerDied","Data":"fa77af55adf74b529026c7303efdcc8809b16aa3fddfd56490cad1469b770121"} Sep 10 13:52:19.540147 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:52:19.540064 2557 scope.go:117] "RemoveContainer" containerID="62f2ac5b38653d7dd228b5437aee9309732ed131c87360d3336976968716f663" Sep 10 13:52:19.549109 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:52:19.549095 2557 scope.go:117] "RemoveContainer" containerID="62f2ac5b38653d7dd228b5437aee9309732ed131c87360d3336976968716f663" Sep 10 13:52:19.549359 ip-10-0-139-174 kubenswrapper[2557]: E0910 13:52:19.549340 2557 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"62f2ac5b38653d7dd228b5437aee9309732ed131c87360d3336976968716f663\": container with ID starting with 62f2ac5b38653d7dd228b5437aee9309732ed131c87360d3336976968716f663 not found: ID does not exist" containerID="62f2ac5b38653d7dd228b5437aee9309732ed131c87360d3336976968716f663" Sep 10 13:52:19.549402 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:52:19.549368 2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"62f2ac5b38653d7dd228b5437aee9309732ed131c87360d3336976968716f663"} err="failed to get container status \"62f2ac5b38653d7dd228b5437aee9309732ed131c87360d3336976968716f663\": rpc error: code = NotFound desc = could not find container \"62f2ac5b38653d7dd228b5437aee9309732ed131c87360d3336976968716f663\": container with ID starting with 62f2ac5b38653d7dd228b5437aee9309732ed131c87360d3336976968716f663 not found: ID does not exist" Sep 10 13:52:19.565274 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:52:19.565252 2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-85b6f7bf49-rlbfc"] Sep 10 13:52:19.570145 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:52:19.570126 2557 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-85b6f7bf49-rlbfc"] Sep 10 13:52:19.612727 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:52:19.612707 2557 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jql82\" (UniqueName: \"kubernetes.io/projected/9b8c43f1-3245-48ff-ab59-3e38e74ab93f-kube-api-access-jql82\") on node \"ip-10-0-139-174.ec2.internal\" DevicePath \"\"" Sep 10 13:52:19.612727 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:52:19.612728 2557 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9b8c43f1-3245-48ff-ab59-3e38e74ab93f-maas-api-tls\") on node \"ip-10-0-139-174.ec2.internal\" DevicePath \"\"" Sep 10 13:52:19.612841 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:52:19.612737 2557 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9b8c43f1-3245-48ff-ab59-3e38e74ab93f-metrics-certs\") on node \"ip-10-0-139-174.ec2.internal\" DevicePath \"\"" Sep 10 13:52:19.612841 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:52:19.612745 2557 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9b8c43f1-3245-48ff-ab59-3e38e74ab93f-openshift-service-ca\") on node \"ip-10-0-139-174.ec2.internal\" DevicePath \"\"" Sep 10 13:52:20.220088 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:52:20.220059 2557 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9b8c43f1-3245-48ff-ab59-3e38e74ab93f" path="/var/lib/kubelet/pods/9b8c43f1-3245-48ff-ab59-3e38e74ab93f/volumes" Sep 10 13:52:20.351338 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:52:20.351307 2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-92n4q_720acda6-0b15-49ac-ab4e-60de70c4324a/ovn-acl-logging/0.log" Sep 10 13:52:20.359959 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:52:20.359939 2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-92n4q_720acda6-0b15-49ac-ab4e-60de70c4324a/ovn-acl-logging/0.log" Sep 10 13:52:20.673704 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:52:20.673615 2557 scope.go:117] "RemoveContainer" containerID="bfa12f90b9a3d411e24ebfa22eccbc75d3e14b860dbd0a1327dbf7c8a72b9668" Sep 10 13:52:20.682125 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:52:20.682107 2557 scope.go:117] "RemoveContainer" containerID="03c9b9dc0293970d73551f6477668cb378a778d21c0f552fb48868c339c5e569" Sep 10 13:52:48.820994 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:52:48.820967 2557 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-7878d84c4d-wmfmn_83246f1d-5fd7-4ae8-95b2-b94d9660b31a/manager/0.log" Sep 10 13:52:48.955770 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:52:48.955740 2557 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_model-serving-api-54d89bd5fc-fc5qw_05ac257e-0d9e-4dfa-9309-a0d065304581/server/0.log" Sep 10 13:52:49.083940 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:52:49.083861 2557 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-7998c67dc4-9rt5c_f7115eea-d966-4e45-b0d4-ab798ec09ef2/manager/0.log" Sep 10 13:52:49.204735 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:52:49.204713 2557 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-699db476d-5nldv_ba4fa399-bbb1-4d97-9b2c-4d19667904c4/copy-manifests/0.log" Sep 10 13:52:49.212907 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:52:49.212876 2557 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-699db476d-5nldv_ba4fa399-bbb1-4d97-9b2c-4d19667904c4/manager/0.log" Sep 10 13:52:50.238677 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:52:50.238645 2557 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-key-cleanup-29817465-jkfgf_070db958-b00e-4b95-a0f5-751dbedf8833/cleanup/0.log" Sep 10 13:52:52.358029 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:52:52.357981 2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-68cf4d799cnxbst_4e2d7c7f-f21c-41ec-aaf6-15d3976f0888/istio-proxy/0.log" Sep 10 13:52:52.489041 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:52:52.489020 2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-5677ccb49b-tkt56_d978ed31-3a9b-4efb-b5c9-da53484bebc9/discovery/0.log" Sep 10 13:52:52.870123 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:52:52.870091 2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-5ccb968c66-hc5bh_c933115e-f7d6-4878-8d5f-a4c81447652d/istio-proxy/1.log" Sep 10 13:52:53.150622 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:52:53.150533 2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5b54b7fbb7-t2dzv_68b5cdbd-ff82-4dc9-aa29-c592a12629e0/payload-processing/0.log" Sep 10 13:52:53.635939 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:52:53.635889 2557 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-distinct-2-simulated-kserve-67d8d5d6d-jg757_a8295d66-90d3-4610-adfe-1e6c03c04a0d/main/0.log" Sep 10 13:52:54.001646 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:52:54.001619 2557 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-trlp-test-simulated-kserve-598466548d-q7pvs_ae1519ac-d912-4bf6-b1bf-5b0a588899f0/main/0.log" Sep 10 13:53:04.036679 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:53:04.036645 2557 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-phs5r_d873f3a0-ccac-4d59-a6f6-2c93cf5118de/global-pull-secret-syncer/0.log" Sep 10 13:53:04.194402 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:53:04.194375 2557 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-kfhgq_a13f63dc-c50e-4f46-84fe-d9718d8b2308/konnectivity-agent/0.log" Sep 10 13:53:04.281284 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:53:04.281254 2557 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-139-174.ec2.internal_b2d5d97b9a9b3ad7edee5cc4d2502cd5/haproxy/0.log" Sep 10 13:53:10.436593 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:53:10.436562 2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_c07cb7c8-c14f-44ca-8b96-118bdcef762f/alertmanager/0.log" Sep 10 13:53:10.507433 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:53:10.507389 2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_c07cb7c8-c14f-44ca-8b96-118bdcef762f/config-reloader/0.log" Sep 10 13:53:10.547086 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:53:10.547060 2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_c07cb7c8-c14f-44ca-8b96-118bdcef762f/kube-rbac-proxy-web/0.log" Sep 10 13:53:10.583034 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:53:10.583003 2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_c07cb7c8-c14f-44ca-8b96-118bdcef762f/kube-rbac-proxy/0.log" Sep 10 13:53:10.620515 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:53:10.620493 2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_c07cb7c8-c14f-44ca-8b96-118bdcef762f/kube-rbac-proxy-metric/0.log" Sep 10 13:53:10.654372 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:53:10.654350 2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_c07cb7c8-c14f-44ca-8b96-118bdcef762f/prom-label-proxy/0.log" Sep 10 13:53:10.699636 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:53:10.699615 2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_c07cb7c8-c14f-44ca-8b96-118bdcef762f/init-config-reloader/0.log" Sep 10 13:53:10.789638 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:53:10.789617 2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-74cb57df7-dtddc_9eea609d-ef83-4cdd-aacb-d0719e24b978/kube-state-metrics/0.log" Sep 10 13:53:10.836033 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:53:10.836007 2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-74cb57df7-dtddc_9eea609d-ef83-4cdd-aacb-d0719e24b978/kube-rbac-proxy-main/0.log" Sep 10 13:53:10.897704 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:53:10.897683 2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-74cb57df7-dtddc_9eea609d-ef83-4cdd-aacb-d0719e24b978/kube-rbac-proxy-self/0.log" Sep 10 13:53:10.963960 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:53:10.963908 2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_metrics-server-6b596c7f59-v6hk8_9543ae87-983f-4d7a-a2f6-36aaf9caca18/metrics-server/0.log" Sep 10 13:53:11.268327 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:53:11.268233 2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-pvp5f_fee71c0c-b8c8-4802-b2dd-e8c2ba7389cc/node-exporter/0.log" Sep 10 13:53:11.302467 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:53:11.302446 2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-pvp5f_fee71c0c-b8c8-4802-b2dd-e8c2ba7389cc/kube-rbac-proxy/0.log" Sep 10 13:53:11.334254 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:53:11.334236 2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-pvp5f_fee71c0c-b8c8-4802-b2dd-e8c2ba7389cc/init-textfile/0.log" Sep 10 13:53:11.381163 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:53:11.381145 2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-657fd6c54-vnznb_f42e5bbe-1bd2-472a-b55d-786b3a65fba9/kube-rbac-proxy-main/0.log" Sep 10 13:53:11.430758 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:53:11.430739 2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-657fd6c54-vnznb_f42e5bbe-1bd2-472a-b55d-786b3a65fba9/kube-rbac-proxy-self/0.log" Sep 10 13:53:11.472212 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:53:11.472195 2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-657fd6c54-vnznb_f42e5bbe-1bd2-472a-b55d-786b3a65fba9/openshift-state-metrics/0.log" Sep 10 13:53:11.976507 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:53:11.976479 2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-6ccc84c476-zwhn8_222b4b70-7a97-4b6e-9f37-840e45dab882/telemeter-client/0.log" Sep 10 13:53:12.019442 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:53:12.019402 2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-6ccc84c476-zwhn8_222b4b70-7a97-4b6e-9f37-840e45dab882/reload/0.log" Sep 10 13:53:12.054106 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:53:12.054084 2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-6ccc84c476-zwhn8_222b4b70-7a97-4b6e-9f37-840e45dab882/kube-rbac-proxy/0.log" Sep 10 13:53:14.198313 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:53:14.198280 2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-9tqt4/perf-node-gather-daemonset-px8vq"] Sep 10 13:53:14.198872 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:53:14.198851 2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9b8c43f1-3245-48ff-ab59-3e38e74ab93f" containerName="maas-api" Sep 10 13:53:14.198963 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:53:14.198876 2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="9b8c43f1-3245-48ff-ab59-3e38e74ab93f" containerName="maas-api" Sep 10 13:53:14.199031 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:53:14.198968 2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="9b8c43f1-3245-48ff-ab59-3e38e74ab93f" containerName="maas-api" Sep 10 13:53:14.202284 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:53:14.202263 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-9tqt4/perf-node-gather-daemonset-px8vq" Sep 10 13:53:14.205130 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:53:14.205107 2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-9tqt4\"/\"openshift-service-ca.crt\"" Sep 10 13:53:14.205265 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:53:14.205168 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-9tqt4\"/\"default-dockercfg-hkk72\"" Sep 10 13:53:14.205576 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:53:14.205563 2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-9tqt4\"/\"kube-root-ca.crt\"" Sep 10 13:53:14.230135 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:53:14.230107 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-9tqt4/perf-node-gather-daemonset-px8vq"] Sep 10 13:53:14.261209 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:53:14.261183 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/5baedea2-64e5-4874-949a-bd265d11f276-lib-modules\") pod \"perf-node-gather-daemonset-px8vq\" (UID: \"5baedea2-64e5-4874-949a-bd265d11f276\") " pod="openshift-must-gather-9tqt4/perf-node-gather-daemonset-px8vq" Sep 10 13:53:14.261209 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:53:14.261211 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/5baedea2-64e5-4874-949a-bd265d11f276-proc\") pod \"perf-node-gather-daemonset-px8vq\" (UID: \"5baedea2-64e5-4874-949a-bd265d11f276\") " pod="openshift-must-gather-9tqt4/perf-node-gather-daemonset-px8vq" Sep 10 13:53:14.261378 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:53:14.261233 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v5gjq\" (UniqueName: \"kubernetes.io/projected/5baedea2-64e5-4874-949a-bd265d11f276-kube-api-access-v5gjq\") pod \"perf-node-gather-daemonset-px8vq\" (UID: \"5baedea2-64e5-4874-949a-bd265d11f276\") " pod="openshift-must-gather-9tqt4/perf-node-gather-daemonset-px8vq" Sep 10 13:53:14.261378 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:53:14.261362 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/5baedea2-64e5-4874-949a-bd265d11f276-podres\") pod \"perf-node-gather-daemonset-px8vq\" (UID: \"5baedea2-64e5-4874-949a-bd265d11f276\") " pod="openshift-must-gather-9tqt4/perf-node-gather-daemonset-px8vq" Sep 10 13:53:14.261485 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:53:14.261395 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/5baedea2-64e5-4874-949a-bd265d11f276-sys\") pod \"perf-node-gather-daemonset-px8vq\" (UID: \"5baedea2-64e5-4874-949a-bd265d11f276\") " pod="openshift-must-gather-9tqt4/perf-node-gather-daemonset-px8vq" Sep 10 13:53:14.361967 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:53:14.361938 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/5baedea2-64e5-4874-949a-bd265d11f276-lib-modules\") pod \"perf-node-gather-daemonset-px8vq\" (UID: \"5baedea2-64e5-4874-949a-bd265d11f276\") " pod="openshift-must-gather-9tqt4/perf-node-gather-daemonset-px8vq" Sep 10 13:53:14.361967 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:53:14.361965 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/5baedea2-64e5-4874-949a-bd265d11f276-proc\") pod \"perf-node-gather-daemonset-px8vq\" (UID: \"5baedea2-64e5-4874-949a-bd265d11f276\") " pod="openshift-must-gather-9tqt4/perf-node-gather-daemonset-px8vq" Sep 10 13:53:14.362168 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:53:14.361987 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v5gjq\" (UniqueName: \"kubernetes.io/projected/5baedea2-64e5-4874-949a-bd265d11f276-kube-api-access-v5gjq\") pod \"perf-node-gather-daemonset-px8vq\" (UID: \"5baedea2-64e5-4874-949a-bd265d11f276\") " pod="openshift-must-gather-9tqt4/perf-node-gather-daemonset-px8vq" Sep 10 13:53:14.362168 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:53:14.362050 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/5baedea2-64e5-4874-949a-bd265d11f276-podres\") pod \"perf-node-gather-daemonset-px8vq\" (UID: \"5baedea2-64e5-4874-949a-bd265d11f276\") " pod="openshift-must-gather-9tqt4/perf-node-gather-daemonset-px8vq" Sep 10 13:53:14.362168 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:53:14.362071 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/5baedea2-64e5-4874-949a-bd265d11f276-sys\") pod \"perf-node-gather-daemonset-px8vq\" (UID: \"5baedea2-64e5-4874-949a-bd265d11f276\") " pod="openshift-must-gather-9tqt4/perf-node-gather-daemonset-px8vq" Sep 10 13:53:14.362168 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:53:14.362078 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/5baedea2-64e5-4874-949a-bd265d11f276-proc\") pod \"perf-node-gather-daemonset-px8vq\" (UID: \"5baedea2-64e5-4874-949a-bd265d11f276\") " pod="openshift-must-gather-9tqt4/perf-node-gather-daemonset-px8vq" Sep 10 13:53:14.362168 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:53:14.362127 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/5baedea2-64e5-4874-949a-bd265d11f276-lib-modules\") pod \"perf-node-gather-daemonset-px8vq\" (UID: \"5baedea2-64e5-4874-949a-bd265d11f276\") " pod="openshift-must-gather-9tqt4/perf-node-gather-daemonset-px8vq" Sep 10 13:53:14.362346 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:53:14.362216 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/5baedea2-64e5-4874-949a-bd265d11f276-sys\") pod \"perf-node-gather-daemonset-px8vq\" (UID: \"5baedea2-64e5-4874-949a-bd265d11f276\") " pod="openshift-must-gather-9tqt4/perf-node-gather-daemonset-px8vq" Sep 10 13:53:14.362346 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:53:14.362223 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/5baedea2-64e5-4874-949a-bd265d11f276-podres\") pod \"perf-node-gather-daemonset-px8vq\" (UID: \"5baedea2-64e5-4874-949a-bd265d11f276\") " pod="openshift-must-gather-9tqt4/perf-node-gather-daemonset-px8vq" Sep 10 13:53:14.377552 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:53:14.377530 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-v5gjq\" (UniqueName: \"kubernetes.io/projected/5baedea2-64e5-4874-949a-bd265d11f276-kube-api-access-v5gjq\") pod \"perf-node-gather-daemonset-px8vq\" (UID: \"5baedea2-64e5-4874-949a-bd265d11f276\") " pod="openshift-must-gather-9tqt4/perf-node-gather-daemonset-px8vq" Sep 10 13:53:14.511998 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:53:14.511942 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-9tqt4/perf-node-gather-daemonset-px8vq" Sep 10 13:53:14.790619 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:53:14.790595 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-9tqt4/perf-node-gather-daemonset-px8vq"] Sep 10 13:53:14.792740 ip-10-0-139-174 kubenswrapper[2557]: W0910 13:53:14.792716 2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod5baedea2_64e5_4874_949a_bd265d11f276.slice/crio-b492670bb3a334b1a2cafb10527282798683dda7e5c7e5fecc825dc462503cc8 WatchSource:0}: Error finding container b492670bb3a334b1a2cafb10527282798683dda7e5c7e5fecc825dc462503cc8: Status 404 returned error can't find the container with id b492670bb3a334b1a2cafb10527282798683dda7e5c7e5fecc825dc462503cc8 Sep 10 13:53:14.826612 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:53:14.826579 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-9tqt4/perf-node-gather-daemonset-px8vq" event={"ID":"5baedea2-64e5-4874-949a-bd265d11f276","Type":"ContainerStarted","Data":"b492670bb3a334b1a2cafb10527282798683dda7e5c7e5fecc825dc462503cc8"} Sep 10 13:53:15.691430 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:53:15.691399 2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_console-6868d857cb-kcx5x_e77492b2-9db5-4716-a2b4-78fcf9011686/console/0.log" Sep 10 13:53:15.833493 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:53:15.833460 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-9tqt4/perf-node-gather-daemonset-px8vq" event={"ID":"5baedea2-64e5-4874-949a-bd265d11f276","Type":"ContainerStarted","Data":"9ca6bab1179bf775d4342f356b10cdf44d118a2fd8ccb6e5133e07e1318ce5cb"} Sep 10 13:53:15.833676 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:53:15.833527 2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-9tqt4/perf-node-gather-daemonset-px8vq" Sep 10 13:53:15.857520 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:53:15.857473 2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-9tqt4/perf-node-gather-daemonset-px8vq" podStartSLOduration=1.857454884 podStartE2EDuration="1.857454884s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-10 13:53:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 13:53:15.853254014 +0000 UTC m=+2156.160248426" watchObservedRunningTime="2026-09-10 13:53:15.857454884 +0000 UTC m=+2156.164449297" Sep 10 13:53:17.885269 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:53:17.885234 2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-9x5xs_a5ede13b-c926-4b47-822c-384dc218e8f5/dns/0.log" Sep 10 13:53:17.928883 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:53:17.928863 2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-9x5xs_a5ede13b-c926-4b47-822c-384dc218e8f5/kube-rbac-proxy/0.log" Sep 10 13:53:18.285180 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:53:18.285154 2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-6wf8v_f2904741-875c-40e6-a97e-e56ae66d5077/dns-node-resolver/0.log" Sep 10 13:53:19.091451 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:53:19.091407 2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-zzkks_9e7f9da8-1931-42da-b5c4-67542de73659/node-ca/0.log" Sep 10 13:53:20.429233 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:53:20.429207 2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-68cf4d799cnxbst_4e2d7c7f-f21c-41ec-aaf6-15d3976f0888/istio-proxy/0.log" Sep 10 13:53:20.677673 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:53:20.677645 2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-5677ccb49b-tkt56_d978ed31-3a9b-4efb-b5c9-da53484bebc9/discovery/0.log" Sep 10 13:53:20.798123 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:53:20.798100 2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-5ccb968c66-hc5bh_c933115e-f7d6-4878-8d5f-a4c81447652d/istio-proxy/0.log" Sep 10 13:53:20.805634 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:53:20.805615 2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-5ccb968c66-hc5bh_c933115e-f7d6-4878-8d5f-a4c81447652d/istio-proxy/1.log" Sep 10 13:53:20.895856 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:53:20.895835 2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5b54b7fbb7-t2dzv_68b5cdbd-ff82-4dc9-aa29-c592a12629e0/payload-processing/0.log" Sep 10 13:53:21.848336 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:53:21.848304 2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-9tqt4/perf-node-gather-daemonset-px8vq" Sep 10 13:53:21.981561 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:53:21.981536 2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-m7dz6_2c8e9d2b-dd0d-4715-b367-54121669c328/serve-healthcheck-canary/0.log" Sep 10 13:53:22.933749 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:53:22.933710 2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-7k285_67184dde-9271-4b22-adb6-9a94adaf6c57/kube-rbac-proxy/0.log" Sep 10 13:53:22.958338 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:53:22.958318 2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-7k285_67184dde-9271-4b22-adb6-9a94adaf6c57/exporter/0.log" Sep 10 13:53:22.986686 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:53:22.986659 2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-7k285_67184dde-9271-4b22-adb6-9a94adaf6c57/extractor/0.log" Sep 10 13:53:25.226171 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:53:25.226144 2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-68cf4d799cnxbst_4e2d7c7f-f21c-41ec-aaf6-15d3976f0888/istio-proxy/0.log" Sep 10 13:53:25.452401 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:53:25.452370 2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-5677ccb49b-tkt56_d978ed31-3a9b-4efb-b5c9-da53484bebc9/discovery/0.log" Sep 10 13:53:25.518773 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:53:25.518682 2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-5ccb968c66-hc5bh_c933115e-f7d6-4878-8d5f-a4c81447652d/istio-proxy/0.log" Sep 10 13:53:25.525671 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:53:25.525652 2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-5ccb968c66-hc5bh_c933115e-f7d6-4878-8d5f-a4c81447652d/istio-proxy/1.log" Sep 10 13:53:25.585034 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:53:25.585013 2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5b54b7fbb7-t2dzv_68b5cdbd-ff82-4dc9-aa29-c592a12629e0/payload-processing/0.log" Sep 10 13:53:26.122003 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:53:26.121972 2557 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-7878d84c4d-wmfmn_83246f1d-5fd7-4ae8-95b2-b94d9660b31a/manager/0.log" Sep 10 13:53:26.145403 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:53:26.145381 2557 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_model-serving-api-54d89bd5fc-fc5qw_05ac257e-0d9e-4dfa-9309-a0d065304581/server/0.log" Sep 10 13:53:26.224677 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:53:26.224652 2557 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-7998c67dc4-9rt5c_f7115eea-d966-4e45-b0d4-ab798ec09ef2/manager/0.log" Sep 10 13:53:26.260826 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:53:26.260804 2557 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-699db476d-5nldv_ba4fa399-bbb1-4d97-9b2c-4d19667904c4/manager/0.log" Sep 10 13:53:26.328145 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:53:26.328126 2557 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-699db476d-5nldv_ba4fa399-bbb1-4d97-9b2c-4d19667904c4/copy-manifests/0.log" Sep 10 13:53:27.139196 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:53:27.139173 2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-68cf4d799cnxbst_4e2d7c7f-f21c-41ec-aaf6-15d3976f0888/istio-proxy/0.log" Sep 10 13:53:27.379059 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:53:27.379030 2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-5677ccb49b-tkt56_d978ed31-3a9b-4efb-b5c9-da53484bebc9/discovery/0.log" Sep 10 13:53:27.441841 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:53:27.441770 2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-5ccb968c66-hc5bh_c933115e-f7d6-4878-8d5f-a4c81447652d/istio-proxy/0.log" Sep 10 13:53:27.452827 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:53:27.452806 2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-5ccb968c66-hc5bh_c933115e-f7d6-4878-8d5f-a4c81447652d/istio-proxy/1.log" Sep 10 13:53:27.508105 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:53:27.508084 2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5b54b7fbb7-t2dzv_68b5cdbd-ff82-4dc9-aa29-c592a12629e0/payload-processing/0.log" Sep 10 13:53:27.877209 ip-10-0-139-174 kubenswrapper[2557]: I0910 13:53:27.877187 2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-lws-operator_lws-controller-manager-78dbbc87c8-mf5qs_7920562d-b64e-478f-b502-abb12f5f1d2a/manager/0.log"